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were  abstracted  for  this  document. 
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I.  INTRODUCTION 


This  report  describes  one  of  the  earliest  tasks  under  the  "Study  of  Effective¬ 
ness  of  Infantry  Systems:  TEA,  CTEA  and  Human  Factors  in  Systems  Development 
and  Fielding"  (Contract  No.  MDA903-80-C-0345).  Dunlap  and  Associates,  Inc.,  is 
responsible  for  Task  3  (Systems  Development  and  Evaluation  Technology)  of  t  lat 
contract,  under  subcontract  (No.  05628)  to  the  Mellonics  Systems  Development  Divi-  _ 
sion  of  TifTon  Systems,  Inc.  This  present  report  is  partial  fulfillment  of  Task  3a, 
"Review  of  the  Manned  Systems  Measurement  Literature." 

Published  research  and  other  literature  pertinent  to  systems  and  measurement 
theory  and  practice  were  identified,  acquired,  reviewed  and  annotated  during  this 
task.  The  purpose  of  these  activities  was  to  obtain  a  clear  understanding  of  the 
state-of-the-art  of  manned  systems  measurement.  This  would  indicate  the  necessary 
direction  and  scope  of  the  effort  required  to  expand  and  update  the  Systems  Taxonomy 
Model  (STM)  and  the  components  of  the  Overall  Conceptual  Process  Model  (OCPM). 

Task  3a  was  divided  into  two  subtasks: 

1.  Obtain  Relevant  Literature 

2.  Review  and  Annotate  Literature 

The  methods  used  for  completing  these  tasks  are  discussed  in  the  next  sections. 
Appendices  A  and  B  are  the  bibliographies  themselves,  in  annotated  and  unannotated 
form,  respectively. 


II.  REVIEW  OF  SUBTASKS 


A.  Subtask  3a(l):  Obtain  Relevant  Literature 


The  identification  and  acquisition  of  relevant  manned  systems  measurement 
literature  was  built  on  an  existing  base  of  documentation.  This  base  consisted  of 
the  searches  conducted  by  AR1  of  the  National  Technical  Information  Service 
(NT1S)  and  Defense  Documentation  Center  (DDC),  now  Defense  Technical  Informa¬ 
tion  Center  (DT1C),  data  bases  in  February  1977.  The  Contracting  Officer's  Tech¬ 
nical  Representative  (COTR)  provided  Dunlap  and  Associates,  Inc.,  with  the  strategy 
utilized  in  these  searches  and  also  approximately  200  documents  which  had  been 
obtained  in  that  1977  search.  Key  words  used  in  the  1977  searches  were:  opera¬ 
tional  test  and  evaluation;  performance  standards  measurement;  systems  development, 
test  and  evaluation;  performance  measurement;  systems  human  operators;  measure¬ 
ment  methodology;  systems  and  methodology;  measure  theory;  and  effectiveness 
measurement. 

Dunlap  updated  and  extended  the  AR1  literature  file  by  conducting  searches 
using  the  same  data  bases  and  key  words  to  acquire  new  entries  since  the  original 
search  was  performed  in  1977.  For  the  NT1S  search,  additional  key  words  were 
selected  from  the  candidate  terms  listed  in  Table  1.  In  addition  to  the  NTIS  and 
DTIC  searches,  the  search  was  expanded  to  include  the  PASAR  and  COMPENDEX 
data  bases.  Complete  bibliographic  citations  were  obtained,  including  abstracts 
and  printouts  from  the  four  searches. 

Two  of  these  searches  (NTIS  and  COMPENDEX)  were  conducted  in-house 
using  the  set  of  DIALOG  data  bases  maintained  by  Lockheed's  Palo  Alto  facility. 

The  Defense  Techncial  Information  Center  conducted  the  DT1S  search  and  the 
American  Psychological  Association  performed  the  PASAR  search  under  the  direc¬ 
tion  of  Dunlap  staff  members. 

The  next  step  was  to  determine  the  relevance  of  the  results  of  the  com¬ 
puterized  search.  A  triage  of  the  results  was  performed  and  abstracts  were 
coded  into  one  of  the  following  three  mutually  exclusive  categories:  V  for  "very 
likely  relevant,"  P  for  "possibly  relevant"  and  L  for  "very  likely  not  relevant." 

The  triaging  results  for  each  data  base  are  summarized  in  Table  2.  The  percen¬ 
tages  given  in  the  table  are  approximate.  In  the  "very  likely  relevant"  category, 
there  is  one  duplication  of  a  report  and  that  occurs  in  the  NTIS  and  DTIC  lists. 

Since  there  may  be  duplications  among  the  other  categories  as  well,  the  actual 
totals  may  be  less  than  the  values  indicated  in  the  table. 

A  copy  of  the  triage  ratings  was  submitted  to  the  COTR  and  based  upon 
discussions  with  the  COTR  and  in  view  of  the  volume  of  materials  already  avail¬ 
able,  it  was  decided  to  obtain,  at  the  present  time,  only  those  documents  classed 
as  "very  likely  relevant."  Approximately  50  such  documents  were  acquired. 
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Table  1.  Candidate  Key  Words  for  Literature  Search 

System  (Design,  Analysis,  Effectiveness,  Definition,  Attributes, 
Testing,  Constraints,  Performance) 

Development  (Methods,  Process,  Models,  Technology) 

System  Development  (Methods,  Process,  Models,  Technology) 

Manned  System  Development  (Methods,  Process,  Models, 
Technology) 

Mission  Definition 

Measurement  (Methods,  Process,  Models,  Analysis,  Techniques, 
Standards) 

System  Measurement  (Methods,  Process,  Models) 

Manned  System  Measurement  (Methods,  Process,  Models) 
Analytic  Methods 
Taxonomy  (Models) 

System  Taxonomy  (Models) 

Evaluation  (Methods,  Process,  Models,  Technology,  Criteria, 
Techniques) 

System  Evaluation  (Methods,  Process,  Models,  Technology) 

Manned  System  Evaluation  (Methods,  Process,  Models, 
Technology) 

Cost  Benefit  (Analysis,  Evaluation,  Methods,  Measures) 

Cost  Effectiveness  (Analysis,  Evaluation,  Methods,  Measures) 
Measures  of  Effectiveness 

Performance  (Requirements,  Criteria,  Analysis,  Measures, 
Assessment) 

Test  (Plans,  Planning,  Methods,  Development) 

Effectiveness  (Evaluation,  Testing,  Measures,  Criteria,  Analysis, 
Assessment) 

Proficiency  (Measures,  Measurement) 

Man-Machine  Systems  (Evaluation,  Methods,  Process,  Models, 
Technology,  Assessment) 

Human  Factors  (Analysis,  Evaluation) 

State  of  the  Art 
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Table  2.  Triage  Results  for  Four  Data  Base  Searches 


Data  Bases 

Relevance 

Total 

V 

P 

L 

NTIS 

23 

29 

62 

114 

DTIC 

19 

49 

32 

100 

PASAR 

7 

10 

15 

32 

COMPENDEX 

9 

15 

27 

51 

TOTAL 

58 

103 

136 

297 

PERCENTAGE 

2096 

35% 

45% 

100% 

B.  Subtask  3a(2):  Review  and  Annotate  Literature 


Following  the  steps  described  above,  abstracting  of  the  relevant  material 
began.  The  format  of  literature  annotation/abstracting  conformed  to  the  com¬ 
ponents  of  the  overall  conceptual  process  model.  That  is,  an  abstract  was 
prepared  for  each  document  included  in  the  annotated  bibliography  using  a  stan¬ 
dard  abstracting  form  as  illustrated  in  Figure  1.  This  standard  abstracting  form 
indicates  all  of  the  conceptual  process  model  components  to  which  the  particular 
document  applies;  the  abstract  itself  consists  primarily  of  separate  summaries  of 
the  document's  contents  relevant  to  the  indicated  model  components. 

An  annotated  bibliography  of  the  relevant  literature  is  presented  in  Appendix 
A.  The  bibliography  indicates  all  of  the  documents  obtained  as  a  result  of  the 
search  performed  in  1977  and  those  documents  that  were  identified  as  "very  likely 
relevant"  in  the  new  searches  and  obtained.  A  listing  of  documents,  without  ab¬ 
stracts,  is  presented  in  Appendix  B. 


r  j 


AC  Spark  Plug  Division,  General  Motors  Corporation.  Inertial  guidance  system 
107 A-2 — Category  II,  Personnel  Subsystem  Test  and  Evaluation  (PSTE)  and 
Maintenance,  Logistics,  Reliability  and  Readiness  (MLRR)  test  and 
evaluation — Ob jective  achievement  status  report  (AF  04  (694)-177  AFBM  Exhibit 
60-20A).  Milwaukee,  WI,  15  February  1964.  (AD-829  749). 
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1.  State  of  the  Art  Review  2.2 

of  the  Process 

1.1  General  System 
Measurements 

1.2  System  Taxonomy 

Model  (STM)  2.5 

1.3  Overall  Conceptual 
Process  Model  (CPM) 

2.  Contextual  Components 
of  the  Process 

2.1  System  Definition 

2.2  Mission  Definition 

2.3  Environment  Definition 

2.4  General  Constraints 

2.5  Performance 
Requirements,  Ultimate 

2.6  Performance 
Criteria,  Ultimate 

3.  Analytic  Components 
of  the  Process 

3.1  Practical  Measurable 
Attributes 

3.2  Practical  Attribute 
Measures 

3.3  Performance 
Requirements,  Specific 

3  -  Performance 

Criteria,  Specific 
3-5  Measurement  Procedures 

4.  Planning  Components 
of  the  Process 

4.1  Analytic  Methods 

4.2  Parameter  Determinations 

4.3  Apparatus  for  Testing 

4.4  Personnel  for  Testing 

4.5  Test  Plans 

3.4 

5.  Application  Components 
of  the  Process 

5.1  Test  Execution 

5.2  Data  Analysis 

5.3  Findings  Interpretation 

5.4  Conclusions  and 
Recommendations 


This  report  specified  the  Personnel 
Subsystem  Test  and  Evaluation  (PSTE) 
objectives  for  the  TITAN  II  IGS. 

Two  system  measures  were  developed:  a 
measure  of  system  "adequacy"  and  a  measure 
of  system  "efficiency".  The  former  measure 
subsumed  the  concepts  of  "availability"  and 
"accuracy",  while  the  latter  measure 
included  "expenditure  per  unit  of  output" 
as  the  primary  yardstick. 

The  PSTE  objectives  were  broken  down  into 
three  components  that  involved  weapon 
system  testing  as  it  related  to  human 
engineering,  personnel,  training,  and 
validation  of  technical  publications: 

(1)  Personnel  Performance 

(2)  Safety 

(3)  Technical  Data 

The  MLRR  (Maintenance,  Logistics, 
Reliability,  and  Readiness)  test  and 
evaluation  included  the  collection  and  data 
analysis  items  relevant  to  weapon  system 
testing. 

Operationalization  of  this  concept  included 
the  following  representative  items:  for 
"maintenance",  an  indicator  might  be  the 
determination  of  whether  the  support 
activities  for  missile  and  group  equipment 
maintenance  are  adequate;  the 


6.  Further  Research  Areas 

6.1  Measurement  System 
Limitations 

6.2  Research  Potentials/ 
Priorities 

6.3  Research  Planning 
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ABSTRACT 

"logistics"  concept  includes  spare  parts 
consumption  and  compatibility  requirements; 
"reliability"  involves  collection  of 
failure  data  regarding  all  subsystems; 
"readiness"  involves  an  evaluation  of  the 
interrelated  effects  of  operations, 
maintenance,  and  reliability  on  the 
"in-commission"  rate.  Finally,  weapon 
system  capability  is  determined  by  the 
product  of  alert  readiness  reliability  x 
launch  reliability  x  in-flight  reliability 
x  warhead  reliability. 


5.4  The  main  conclusion  for  the  Personnel 

Subsystem  section  found  personnel  operating 
within  tolerance  limits  although 
contributing  20%  to  the  total  downtime  of 
the  system.  A  morale  problem  was  found  due 
to  the  minimum  level  of  capability  required 
for  the  guidance  system  that  underutilized 
personnel  skills.  Corrective  training  and 
reorientation  were  required. 


Akashi,  H.  A  Mahmood,  S.  Performance  of  human  operators  under  various  system 
parameters  (NASA  CR-6725).  National  Aeronautics  and  Space  Administration, 
(undated) 


I  Topics  Relevant 

1  1 

1  1 

{  to  System  Development 

iTopici 

land  Evaluation  Technology 

1  No.  i 

ABSTRACT 

1.  State  of  the  Art  Review 
of  the  Process 

1.1  General  System 
Measurements 

1.2  System  Taxonomy 
Model  (STM) 

1.3  Overall  Conceptual 
Process  Model  (CPM) 


2.1  A  closed  loop  system,  consisting  of  a 
display,  control  stick,  an  analogue 
computer,  an  operator  and  procedures,  was 
used  in  this  evaluation. 

2.2  The  performance  of  operators  under  various 
system  parameters  was  measured. 


2.  Contextual  Components 

of  the  Process  2 . 3 

2.1  System  Definition 

2.2  Mission  Definition 

2.3  Environment  Definition 

2.4  General  Constraints  3.1 

2.5  Performance 
Requirements,  Ultimate 

2.6  Performance 
Criteria,  Ultimate 

3  2 

3.  Analytic  Components 
of  the  Process 

3.1  Practical  Measurable 
Attributes 

3.2  Practical  Attribute 
Measures 

3.3  Performance  3.3 

Requirements,  Specific 

3.4  Performance 
Criteria,  Specific 

3.5  Measurement  Procedures 

4.3 

4.  Planning  Components 
of  the  Process 

4.1  Analytic  Methods 

4.2  Parameter  Determinations 

4.3  Apparatus  for  Testing  tj  n 

4.4  Personnel  for  Testing 

4.5  Test  Plans 

5.  Application  Components 
of  the  Process 

5.1  Test  Execution 

5.2  Data  Analysis 

5.3  Findings  Interpretation 

5.4  Conclusions  and 
Recommendations 

6.  Further  Research  Areas 

6.1  Measurement  System 
Limitations 

6.2  Research  Potentials/ 

Priorities 

6.3  Research  Planning 


Untrained  operators  were  tested  in  a 
laboratory  situation. 

The  analogue  computer  compared  the  random 
input  signal  with  operator  tracking  signal 
and  computed  an  error  signal. 

A  performance  index  which  is  the  fraction 
of  the  total  time  during  which  the  error 
signal  exceeded  an  arbitrarily  chosen 
threshold  was  used  as  the  measure. 

The  objective  was  to  measure  the 
performance  of  human  operators  under 
various  system  parameters. 

A  display  (oscilloscope),  control  stick, 
and  a  controlled  element  (analogue 
computer)  were  used  for  this  study. 

Operators  were  taken  from  3  groups: 

(1)  persons  with  both  licensed  flying  and 
driving  experience;  (2)  persons  with 
average  driving  experience;  (3) 
non-drivers.  Ages:  20-40. 
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5.3  The  results  show  that  the  performance  of 
operators  can  be  represented  in  the 
parameter  plane  of  time  constant  and  gain 
by  a  hyperbolic  curve,  its  distance  from 
the  origin  showing  the  control  ability  of 
the  operator  and  its  general  shape 
indicating  the  operator's  adaptability  to 
the  variation  of  the  two  parameters. 

5.^  The  effect  of  changing  the  nature  of  random 
noise,  the  display  device  and  the  manual 
control  have  yet  to  be  studied. 


Analytics,  Inc.  Measuring  the  performance  of  operational  decision  aids 
(N00014-75-C-0600,  Final  Rep.  1161-B).  Willow  Grove,  PA:  Analytics,  Inc., 
April  1976.  (AD-A024  7  95). 
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State  of  the  Art  Review 
of  the  Process 

1.1  General  System 
Measurements 

1.2  System  Taxonomy 
Model  (STM) 

1.3  Overall  Conceptual 
Process  Model  (CPM) 

Contextual  Components 
of  the  Process 

2.1  System  Definition 

2.2  Mission  Definition 

2.3  Environment  Definition 

2.4  General  Constraints 

2.5  Performance 
Requirements,  Ultimate 

2.6  Performance 
Criteria,  Ultimate 

Analytic  Components 
of  the  Process 

3.1  Practical  Measurable 
Attributes 

3-2  Practical  Attribute 
Measures 

3-3  Performance 

Requirements,  Specific 
3  •  4  Performance 

Criteria,  Specific 
3-5  Measurement  Procedures 

Planning  Components 
of  the  Process 

4.1  Analytic  Methods 

4.2  Parameter  Determinations 

4.3  Apparatus  for  Testing 

4.4  Personnel  for  Testing 

4.5  Test  Plans 


1.1  The  problem  addressed  is  the  development  of 
a  "methodology"  for  testing  the 
acceptability  of  Navy  operational  decision 
aids  (or  systems).  Resulting  tests  must  be 
scientifically  sound,  statistically 
reproducible,  objective  and  unbiased, 
extrapolatable  to  real  world  conditions, 
plausible,  defensible,  and  evaluatable  in 
terms  of  liabilities,  performance  and  risk. 
The  test  methodology  must  assess  the 
functional  performance  of  the  system  and 
cannot  be  designed  to  match  the  system 
itself,  lest  the  result  be  determined  by  the 
evaluation  method. 

3.1  Classes  of  systems  were  expected  to  be 
identified,  such  as  information  storage  and 
retrieval  systems.  This  methodological 
study  addressed  the  formulation  of  measures 
of  performance  (MOP's)  applicable  to 
information  storage  and  retrieval  type 
systems.  Two  types  of  measures  emerged: 

(1)  asymptotic  measure  of  improvement  in  the 
decision  process,  and  (2)  a  time  constant 
identifying  the  rate  of  improvement.  The 
measures  are  calculable  from  experimental 
measures  of  reliance,  irrelevance  and  time. 
The  specific  and  tentative  measures  are 
shown  to  leave  a  common  interpretation  as 
Bayesian  updated  probabilities. 
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2.5  The  utility  of  a  tactical  interdiction 
aircraft  depends  upon  kill  potential, 
probability  of  reaching  the  target, 
probability  of  survival,  and  availability. 
The  aircraft's  worth  cannot  be  assessed  by 
considering  these  factors  in  isolation. 
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Measures  of  effectiveness  must  be  developed 
which  quantitatively  account  for  the 
interaction  of  these  characteristic 
effectiveness  parameters.  Any  valid 
measure  of  effectiveness  must  account  for 
the  cumulative  effect  of  repeated  sorties. 

To  obtain  a  measure  of  effectiveness  of  an 
aircraft  in  a  given  scenario,  it  seems 
reasonable  to  keep  the  scenario  fixed 
(fixed  characteristic  parameters)  and  to 
determine  cumulative  effectiveness  under 
repeated  sorties  in  that  fixed  scenario. 

The  expected  number  of  targets  destroyed 
was  expressed  as  a  function  of  several 
probablistic  variables,  including  kill 
potential,  probability  of  reaching  target 
and  releasing  weapons,  probability  of 
survival,  etc.  The  expected  number  of 
targets  destroyed  was  computed  for  the 
aircraft’s  total  lifetime  as  well  as  for  an 
arbitrary  number  of  sorties. 
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complete. 
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2.2  The  radar  was  designed  for  a  range  of  up  to 
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detect,  identify,  and  classify  targets 
within  a  defined  air  space  to  the  Tactical 
Air  Operations  Center  (TAOC).  The  purpose 
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analysis  on  the  operational  effectiveness, 
suitability,  and  military  utility  of  the 
Radar  Set. 


3.  Analytic  Cooponents 
of  the  Process 

3. 1  Practical  Measurable 
Attributes 

3.2  Practical  Attribute 
Measures 

3.3  Performance 
Requirements,  Specific 

3- A  Performance 

Criteria,  Specific 
3.5  Measurement  Procedures 

4.  Planning  Cooponents 
of  the  Process 

a . 1  Analytic  Methods 
A.  2  Parameter  Determinations 
A.  3  Apparatus  for  Testing 
A. A  Personnel  for  Testing 
A. 5  Test  Plans 


The  Operational  Testing  (OT)  objectives 
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This  paper  summarizes  an  approach  toward 
developing  a  general  information  system 
model  which  focuses  on  man  and  considers 
the  computer  only  as  a  tool.  The  ultimate 
objective  is  to  produce  a  simulator  which 
will  yield  measures  of  system  performance 
under  different  mixes  of  equipment, 
personnel,  and  procedures.  In  structuring 
the  framework  for  this  model,  the 
assumption  was  made  that  men  have  five 
basic  and  critical  operations  to  perform  in 
an  information  system:  screen,  transform, 
input,  assimilate,  and  decide.  These 
operations,  or  functional  areas,  are 
interrelated  along  three  dimensions:  (1) 
a  data  flow  and  processing  dimension;  (2)  a 
task  analysis  dimension  for  each  event  in 
the  data  flow  sequence;  and  (3)  a  source  of 
variation  dimension,  such  as  level  of 
training.  The  model  approach  described  has 
several  major  points  of  payoff.  Among  the 
immediate  benefits  is  the  potential  for 
using  the  model  to  quantify  human 
performance  by  employing  system  measures 
and  the  value  of  the  model  as  a  tested, 
usable  tool  for  developing  test  and 
evaluation  plans  which  will  provide  human 
factors  data  as  part  of  the  information 
system  design  verification  checkout. 
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The  system  under  evaluation  is  the  highway 
vehicle  in  interactions  with  its  occupants, 
the  terrain  and  physical  obstacles,  during 
the  course  of  ordinary  operations  and  in 
the  event  of  accidents. 

The  parameters  selected  for  measurement 
using  the  air-bearing  device  described  in 
this  study  were  chosen  to  match  those  in  an 
existing  mathematical  simulations  model,  so 
that  later  comparisons  could  serve  to 
validate  the  model. 

The  measurement  equipment,  comprised  of  a 
system  of  air  bearings,  was  designed  to 
help  measure  vehicle  parameters  in  a  manner 
suitable  for  dynamic  studies  of  vehicles. 

By  providing  the  data  needed  in  the 
mathematical  model,  the  measurement  system 
helps  to  facilitate  studies  involving 
various  aspects  of  the  vehicle  occupant 
terrain  obstacle  system.  The  air  bearing 
configuration  of  the  measurement  device  was 
determined  by  the  requirement  for 
flexibility  in  the  types  of  experiments  to 
be  conducted  and  by  the  availability  of 
local  expertise  in  the  technology.  The 
majority  of  this  document  describes  the 
apparatus  and  its  utilization. 
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The  human  element  in  electronic  production 
systems  is  the  focus  of  this  report. 

Reliability  of  human  performance  in  factory 
processes  is  the  issue  of  concern. 

The  measurable  attribute  for  assessing 
reliability  and  the  worker's  contribution 
is  the  rate  of  escape  into  the  field  of 
products  with  unacceptable  workmanship 
defects. 

An  audit  of  final  products,  in  combination 
with  defects  found  by  inspection,  leads  to 
the  calculation  of  an  Estimated  Outgoing 
Quality  Level  (EOQL),  which  is  an  overall 
performance  effectiveness  measure. 

The  specific  requirement  is  to  maintain  an 
overall  Average  Outgoing  Quality  Limit 
(AOQL).  The  overall  AOQL  is  budgeted  to 
establish  contributing  AOQL's  for 
production  subdivisions.  These  figures,  in 
turn,  are  used  to  specify  a  Submitted 
Quality  Level  (SQL),  for  products  submitted 
for  inspection. 

The  study  is  concerned  with  measuring 
output  quality  and  setting  acceptable 
limits  for  poor  workmanship  escapes  and 
product  degradation.  This  includes 
establishment  of  a  3-level  classification 
scheme  for  defects,  based  on  the  expected 
impact  of  defects  on  product  performance. 
Consistent  measurement  also  requires  the 
standardization  of  vocabulary  for 
describing  defects  found  during  the 
inspection  process. 
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This  report  contains  guidelines  for 
conducting,  analyzing  and  reporting  Human 
Factors  Engineering  (HFE)  tests  according 
to  the  specifications  of  DI-H-IBB^A.  The 
requirements  imposed  by  the  specification 
are  presented  together  with  suggested 
sources  of  information. 

A  brief  description  of  the  test  activities 
necessary  to  meet  the  requirements  of 
D1-H-133A4  is  presented  below: 

(1)  Test  administration  —  Includes 
milestone  development,  manpower 
specification  and  budget  preparation. 

(2)  Task  group  description  —  This  task 
requires  task  group  identification 
(all  operations  and  maintenance  tasks 
assigned  to  a  single  personnel 
position);  task  analysis  (defines  in 
detail  the  behavioral  requirements  of 
the  task) ,  and  performance  standards 
identification  (the  identification  of 
the  specific  functions  that  the  system 
must  program). 

(3)  Test  planning  and  design  —  Test 
planning  begins  with  a  statement  of 
test  objectives.  It  is  felt  that  the 
more  precisely  the  test  objectives  are 
defined  the  easier  it  is  to  develop 
the  test  plan.  The  next  task  is  to 
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select  and  design  the  test  equipment. 
The  nature  of  this  configuration 
depends  upon  the  stage  of  system 
development.  Test  environment 
conditions  and  the  need  to  simulate 
those  environmental  conditions  likely 
to  affect  task  group  performances  are 
considered.  After  the  required 
environmental  conditions  have  been 
specified,  provisions  for  measurement 
of  critical  conditions  must  be  made. 
For  example,  changing  environmental 
conditions  will  necessitate  more 
frequent  measurement.  It  is  noted 
that  to  a  large  extent,  the  validity 
of  the  HFE  test  results  depends  on 
selection  of  test  personnel.  There¬ 
fore,  detailed  guidelines  are 
presented  in  the  report  regarding 
personnel  selection  and  personnel 
training. 

With  regard  to  data  acquisition  and 
analysis  planning,  it  is  noted  that 
both  subjective  and  objective  data 
collection  techniques  are  required  to 
meet  the  specifications  of  D1— H—  1 331* A . 
Subjective  data  can  be  collected  by 
such  means  as  ratings,  rankings, 
questionnaires  and  interviews. 

However,  it  is  suggested  that 
objective  measures  be  employed  as  much 
as  possible  to  better  provide 
comparison  of  the  obtained  measures 
and  to  determine  the  degree  to  which 
performance  standards  are  met.  In 
addition,  errors  arising  from  human 
judgment  are  minimized. 

(4)  HFE  test  execution  —  It  is  recom¬ 
mended  that  a  pretest  be  conducted  and 
procedures  are  set  forth  for  conduct¬ 
ing  such  an  activity. 
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(5)  Data  analyses  —  Various  data  analysis 
techniques  and  the  applicability  of 
these  techniques  to  the  type  of  data 
collected  are  discussed. 


(6)  Sunmary  —  A  checklist  of  the 

activities  associated  with  human 
factors  engineering  testing  is 
included  in  this  report. 
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A  technique  was  developed  to  quantify  and 
relate  human  performance  to  the  operability 
component  of  systems  effectiveness.  This 
man-machine  modeling  technique  is  titled 
"Technique  for  Establishing  Personnel 
Performance  Standards  (TEPPS),"  and  two 
field  tests  of  its  application  are  reported 
here. 

A  graphic  mapping  technique  (similar  to  a 
block  diagram  or  flow-chart)  is  employed  to 
show  how  the  system  is  intended  to  operate, 
how  it  can  operate  (unintended),  its 
various  required  operating  states,  and  the 
logic  for  developing  a  conditional 
probability  model.  Similar  to  the 
conventional  reliability  equation,  the 
TEPPS  mathematical  model  is  used  in 
derivative  fashion  to  determine  contri¬ 
buting  probabilities  of  successful  sub¬ 
system  performance  when  the  overall  system 
required  performance  probability 
(reliability)  is  known.  The  model  is  used 
in  integrative  fashion  to  determine  the 
overall  probability  of  successful  perform¬ 
ance  when  the  contributing  subsystem 
probabilities  are  known  or  assumed.  Due  to 
the  lack  of  actual  human  capability  data, 
TEPPS  was  designed  to  accept  relative 
estimates  of  human  capability,  which  are 
obtained  by  subjective  scaling  techniques. 

The  unavailability  of  valid  human 
performance  data  seriously  limits  the 
utility  of  the  model.  Since  the  lack  of 
relevant  performance  data  is  not  likely  to 
change  soon,  subjectively  derived  data 
continues  to  be  given  prime  emphasis.  In 
addition,  there  is  a  need  to  simplify, 
apply  and  test  this  technique  so  that  it 
can  be  made  more  practical  for  evaluating 
the  human  component  in  systems. 
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selection  analysis  (Army  Research  Institute  Research  Note  80-41)  Darien,  CT: 
Dunlap  and  Assoc.,  Inc.,  July  1979. 
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This  is  the  final  report  of  an  analysis  to 
determine  extranormal  selection 
requirements  for  crew  members  of  the 
Infantry  Fighting  Vehicle  (IFV)  and  the 
Cavalry  Fighting  Vehicle  (CFV). 

The  procedures  developed  to  achieve  the 
research  objectives  began  with  a 
clarification  of  objectives  and 
assumptions.  That  clarification  served 
mainly  to  emphasize  the  investigation's 
concern  with  extranormal  attributes  only. 

It  was  then  determined  from  othe: 
concurrent  efforts  that  the  two  vehicles 
(IFV  and  CFV)  were  similar  enough  so  that  a 
single  consolidated  set  of  five  crew 
positions  was  appropriate  ror  this 
analysis:  Track  Commander,  Driver,  Gunner, 
Firing  Port  Weapon  Operator  (IFV  only)  and 
Observer  (CFV  only).  Next,  a  taxonomy  of 
62  personnel  attributes  was  constructed, 
and  a  representative  set  of  IFV/CFV  mission 
scenarios  was  developed.  The  operator's 
task  and  subtask  demands  occurring  during 
exercise  of  the  mission  scenario  were 
analyzed  to  identify  which  of  these 
attributes  in  the  taxonomy  were  required  to 
perform  the  task  or  subtask.  Current 
infantry  and  cavalry  tasks  were  analyzed  to 
determine  the  soldier  attributes  required 
to  perform  the  tasks.  These  attributes 
were  then  compared  with  those  required  to 
perform  the  IFV/CFV  mission  to  identify 
those  attributes  that  were  new  or  unique  to 
IFV/CFV.  Six  potentially  extranormal 
attributes  were  identified  for  the  Track 
Commander  (TC)  and  Gunner  positions,  and 
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three  for  the  Driver  position,  on  the  basis 
that  they  appear  tc  be  new  to  current  MOS 
1 1B  or  19D  personnel.  Those  attributes  are 
especially  needed  to  perform  the  new  or 
unique  IFV/CFV  tasks,  and  they  are  not  now 
used  individually  for  personnel  selection. 
Any  of  the  attributes  is  considered 
extranormal  if  it  must  be  possessed  at  the 
level  of  the  mean  or  higher  so  that  50%  or 
less  of  the  personnel  pool  will  provide  the 
necessary  level. 

4.1  Means,  standard  deviations,  and  ranges  for 
any  subset  of  scores  were  used  in  the 
analyses . 
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Three  modest,  coordinated  efforts  were 
carried  out  to  help  in  providing  better 
techniques  for  the  design,  evaluation,  and 
analysis  of  computer  systems:  (1)  the 
development  of  design  principles  for 
languages  to  model  and  simulate  computer 
systems,  (2)  the  evaluation  and  extension 
of  measuring  and  analyzing  the  performance 
of  complex  computer  systems,  and  (3)  the 
analysis  of  controlled  experiments  in 
man-computer  problem-solving.  The  first 
effort  provided  a  set  of  terms  and  phrases 
described  by  the  author  as  convenient  and 
natural  while  maintaining  the  flexibility 
and  power  of  a  general-purpose  simulation 
language.  The  second  effort  was  a  brief 
review  of  previous  studies  and  a 
description  of  needed  further  studies.  The 
third  effort  reported  on  an  experiment  to 
test  the  effect  of  forced  temporal  lockout 
intervals  on  human  performance  in 
man-computer  problem-solving.  It  concludes 
that  the  relationships  involved  in 
man-machine  problem-solving  are  neither 
obvious  nor  simple,  and  further 
investigation  is  necessary. 

An  appendix  addresses  the  development  of 
performance  criteria  that  are 
discriminating  and  measurable.  It 
describes  a  productive  thought  ratio 
(P.T.R.),  based  on  the  time  spent  thinking 
about  the  project  in  comparison  to  time 
spent  thinking  about  programs  and  waiting 
for  computer  responses. 
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1.1  In  this  Russian  paper,  there  is  a  general 

discussion  of  the  need  to  make  broader  use, 
in  the  practice,  planning  and  management  of 
the  national  economy,  of  novel  methods,  in 
particular  systems  analysis. 

The  history  of  systems  analysis,  its 
essentials,  and  its  application  to  the 
solution  of  problems  in  the  concrete 
sciences  are  discussed.  Presented  are  the 
general  characteristics  which  emerge  in  the 
planning  and  management  of  the  national 
economy  in  its  pi  esent  stage  and  require 
systems  analysis  application. 
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Outlined  is  the  perspective  of  the 
development  of  systems  analysis  into 
systemology,  a  complex  science  of  systems. 
The  role  of  Marxist-Leninist  philosophy  as 
the  methodology  for  further  improvement  and 
development  of  systems  analysis  is  also 
discussed . 
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1.1  The  report  is  concerned  mainly  with  methods 
of  assessing  the  effectiveness  of  training 
programs,  materials,  and  techniques,  with 
special  focus  on  Computer-Aided  and 
Computer-Managed  Instruction  (CAI,  CMI). 
Concern  with  "effectiveness"  of  training 
implies  concern  with  items  of  information 
that  show  how  well  the  teaching  objectives 
are  realized  in  the  students  receiving  the 
treatment  (training). 

1.3  The  basic  assessment/evaluation  methodology 
can  be  outlined  as  follows: 

(1)  A  clear  statement,  in  observable 
terms,  of  the  expected  results  of  the 
treatment,  including  the  time  span 
over  which  a  specific  result  can  be 
measured. 

(2)  Development  of  relevant,  reliable 
yardsticks  (MOEs)  which  measure 
progress  toward  the  stated  objectives 
(expected  results). 
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Application  of  the  yardsticks  within 
the  time  spans  of  the  objectives. 

Establishment  of  an  evaluation  design 
allowing  the  treatment  effects  to  be 
distinguished  from  intervening 
contaminants. 

Establishment  of  the  kinds  and  sources 
of  information  required  to  evaluate 
the  treatment  in  terms  of  the 
objectives. 


ABSTRACT _ 

(6)  Specification  and  examination  of 

underlying  personality  and  situational 
factors  which  explain  the  identified 
change. 

2.6  Only  relatively  few  indices  have  much 

practical  use  as  criteria  for  evaluating 
learning.  These  include: 

(1)  High  degree  of  accuracy  in  performing 
the  learned  response. 

(2)  Significantly  shorter  reaction  latency 
than  at  the  beginning  of  practice. 

(3)  Increased  rate  or  speed  of  correct 
response. 

(M)  Increased  amplitude  of  response. 

(5)  Increased  resistance  to  experimental 
extinction. 

(6)  Increased  resistance  to  retroactive 
inhibition  from  subsequent  learning  as 
compared  to  the  amount  occurring  when 
learning  stops  short  of  mastery. 

(7)  Increased  positive  transfer  to 
subsequent  learning  in  similar 
situations. 

(8)  A  degree  of  generalization  to  similar 
status  events. 

3.2  General  learning  measurements  that  can  be 
applied  to  practical  training  include  the 
following: 

(1)  Gain  scores  (difference  between 
post-test  and  pre-test  scores). 

(2)  Process  scores  (assessment  based  upon 
application  of  procedures  rather  than 
overall  success  in  problem-solving) . 
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(3)  Time  to  criterion  (time  required  to 
complete  some  work  or  achieve  some 
level  of  success) . 

(4)  Error  rate. 

(5)  Persistence  measures  (staying  with 
some  specific  training  sequence). 

(6)  Transfer  measures  (generalizability  of 
the  learning  to  other  situations). 

(7)  Time  vs.  achievement  measures. 

(8)  Retention  measures. 

The  authors  describe  the  strengths  and 
weaknesses  of  these  various  measures,  and 
suggest  certain  types  of  training  programs 
for  which  particular  measures  might  be  most 
applicable. 

4.1  Evaluation  designs  that  are  considered 
applicable  to  assessment  of  training 
effectiveness  include  the  classic  Solomon 
four-group  design;  iterative  adaptation  to 
individual  student  progress;  response 
surface  designs;  adaptive  control  models; 
decision  theory  models;  simulation  models. 

5.4  Principal  conclusions  are  that  the  classic 
four-group  design  is  impractical  for  most 
training  evaluation;  that  "adaptive 
research  for  big  effects"  is  apt  to  be 
scientifically  and  administratively 
desirable;  and  that  current  measurement  of 
training  outcomes  still  uses  fairly  simple 
methods. 
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The  report  describes  a  systems  approach  to 
predicting  and  measuring  the  operational 
availability  of  a  system.  Operational 
availrability,  along  with  the  performance 
capability  level  is  a  major  determinant  of 
the  system’s  operational  effectiveness.  A 
multi-moded  system  is  operationally 
available  whenever  it  is  not  down,  i.e., 
whenever  its  performance  equals  or  exceeds 
the  required  level. 

A  multi-moded  system  is  described  as  a 
collection  of  functionally  connected  but 
independent  subsystems.  Each  subsystem  is 
a  set  of  identical  functional  groups  of  a 
given  type. 

This  viewpoint  of  systems  and  subsystems 
permits  a  simple  mathematical  prediction  of 
the  expected  operational  availability  of 
the  system  at  each  performanc|hlevel.  The 
operational  availability  of  i  subsystem 
is  shown  to  be  a  function  of:  probability 
that  any  functional  group  in  the  i 
subsystem  is  non-failed  at  a  random  point 
in  time;,  nunber  of  such  functional  groups 
in  the  itn  subsystem;  minimum  number  of 
non-failed  functional  groups  required  for 
the  ith  subsystem  to  operate  in  the 
particular  mode  in  question.  The  total 
system  operational  availability  is  the 
product  of  the  subsystem  availabilities. 
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1.3  This  report  summarizes  the  results  of  the 
initial  phase  of  a  three  phase  study.  The 
study  is  designed  to  provide  (a)  instruct¬ 
ional  goals  for  each  program  that  are 
stated  in  measurable  terms  and  (b)  corre¬ 
sponding  performance-based  Go/No  Go  test 
items  suitable  for  evaluating  trainee 
achievement  of  these  goals. 

2.1  The  systems  addressed  were  the  armor 
crewman  and  reconnaissance  specialist 
Advanced  Individual  Training  programs 
(AIT). 

3.1  In  this  phase,  job  related  tasks  addressed 
in  the  AIT  programs  were  examined  and 
tentative  proficiency  levels  were 
established. 

3.2  All  tasks  were  individually  reviewed  and 
coded  to  reflect  the  estimated  level  of 
mastery  required  at  the  end  of  training  and 
prior  to  job  entry. 

3.5  Tasks  were  coded  into  one  of  four 

categories  which  represent  degrees  of 
proficiency  ranging  from  complete 
qualification  to  basic  orientation. 
Definitive  Go/No  Go  performance  criteria 
were  established  for  those  requirements 
considered  relevant  to  job  entry. 

5.4  Data  collected  during  this  first  phase  will 
be  used  as  a  basis  for  formulating 
performance  training  objectives  and  tests 
to  be  subsequently  incorporated  in  new  Army 
Subject  Schedules. 


Breaux,  R.  Training  characteristics  of  the  automated  adaptive  Ground 
Controlled  Approach  Radar  Controller  Training  System  (GCA-CTS) 
(NAVTRAEQUIPCEN-TN-52).  Orlando,  FL:  Naval  Training  Device  Center,  July  1976. 
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2.2  The  automated  adaptive  controller  system 

had,  as  its  primary  objectives,  an  increase 
in  training  effectiveness  by  application  of 
automated  performance  measurement, 
self-paced  instruction,  and  a  type  of 
adaptive  training  oriented  around  cognitive 
skills  development. 


2.  Contextual  Components 

of  the  Process  3.1  The  risk  or  problem  areas  outlined  in  the 

2.1  System  Definition  system  design  included  the  following: 

2.2  Mission  Definition  ' 

2.3  Environment  Definition 
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Performance  Measurement:  to  evaluate 
trainee  performance,  frequency  counts 
of  errors  were  made  in  each  category; 
error  frequency  counts  are  then 
combined  in  a  linear  combination  and 
weighted  to  produce  a  single  composite 
score. 

Adaptive  Logic:  the  problem  here  is 
determining  the  sequence  of  problems 
within  the  automated  performance 
measurement  system;  the  assumption  of 
the  arrangement  of  the  syllabus  is 
based  on  an  increasingly  linear 
difficulty  of  problems. 

Adaptive  Variables:  the  arrangement 
of  the  syllabus  in  terms  of  increasing 
difficulty  is  based  on  various 
adaptive  variables  such  as  wind 
factors,  aircraft  type,  pilot 
response,  and  pilot  variability. 

Student  Feedback:  the  trainee  is 
given  knowledge  of  his  results  at  the 
end  of  each  "run." 
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3.2  Measures  were  made  on  system  output 
variables  and  on  control  input  variables. 
The  former  referred  to  the  simulated 
history  of  the  aircraft  around  the 
glidepath  and  runway  centerline;  the  latter 
consisted  of  trainee  behavior  measures, 
e.g.,  elapsed  time  between  advisories. 

A  single  composite  score  was  created  by 
weighting  a  combination  of  these  measures. 
The  difficulty  level  of  the  next  problem 
was  then  selected  using  this  score  on  the 
adaptive  logic. 

5.3  The  laboratory  feasibility  model  showed 
and  that  the  trainee  must  be  consistent  in  his 

5.4  speaking  voice  before  voice  reference 
pattern  data  is  collected. 

With  regard  to  the  training,  the  key  to 
effective  teaching  was  found  in  the  order 
or  sequence  of  the  problems:  tasks  were 
systematically  introduced  in  ascending 
order  of  complexity. 


Brown,  D.  Statistical  guide  for  CDEC  experimental  design  (Rev.  Preliminary 
Draft).  BDM  Scientific  Support  Laboratory,  October  1976. 
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1.1  This  document  provides  selected  experimen¬ 
tal  descriptors  which  can  be  used  to 
estimate  sample  size  requirements  and 
compare  potential  design  schemes  during  the 
planning  phase  of  military  field  tests. 

The  data  is  compiled  and  organized  under 
six  general  categories:  intervisibility, 
detection,  identification,  localization/pin¬ 
point,  engagement,  probability  of  hit/kill. 

1.2  Appendix  B  provides  a  listing  of  measures 
and  of  effectiveness  by  the  following  experiment 

3.2  types: 

(1)  Mounted  Unit  Operations. 


3.  Analytic  Components 
of  the  Process 

3.1  Practical  Measurable 
Attributes 

3-2  Practical  Attribute 
Measures 

3.3  Performance 

Requirements,  Specific  (2) 

3.4  Performance 
Criteria,  Specific 

3.5  Measurement  Procedures 

4.  Planning  Components 
of  the  Process 

4.1  Analytic  Methods 

4.2  Parameter  Determinations 

4.3  Apparatus  for  Testing 

4.4  Personnel  for  Testing 

4.5  Test  Plans  ^ 

5.  Application  Components 
of  the  Process 

5. 1  Test  Execution 

5.2  Data  Analysis 

5.3  Findings  Interpretation 

5.4  Conclusions  and 
Recommendations 


a.  Mounted  Unit 
Organization/Employment 

b.  Antitank  Weapors  Fire  Effect 
Dismounted  Combat  Operations. 

a.  Dismounted  Unit 
Organization/Employment 

b.  Small  Arms  Effectiveness 

c.  Detection 

Indirect  Fire  Support. 

a.  Forward  Observer  and  Gunner 
Operations 


6.  Further  Rerearch  Areas 

6.1  Measurement  System 
Limitations 

6.2  Research  Potentials/ 
Priorities 

6.3  Research  Planning 
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b.  Gunnery  Operations 

(4)  Army  Aircraft  Operations. 

a.  Air  System-Ground  Target 

b.  Ground  System-Air  Target 

c.  Air  System-Air  Target 

d.  Aircraft  Operations 

(5)  Special. 

a.  Combat  Support  Operations 

b.  Night  Operations 

c.  Line  of  Sight/Exposure 

d.  Two-Sided  Experiments 
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2.1  The  purpose  of  this  study  was  to  assess  the 
operational  effectiveness  of  the  product 
improved  items  on  the  M48A5  tank  to  include 
crew  duties  and  maintenance. 

The  selected  product  improvements  included 
a  low  profile  commanders'  cupola,  extern¬ 
ally  mounted  7.62  mm  machine  guns  and  a 
redesigned  54-round  main  gun  ammunition 
storage  rack. 

2.3  The  tests  were  conducted  with  os  much 

tactical  realism  as  possible  and  included 
operation  on  primary,  secondary,  and 
cross-country  terrain. 


3.  Analytic  Components 

of  the  Process  3.1 

3.1  Practical  Measurable 
Attributes 

3.2  Practical  Attribute 
Measures 

3.3  Performance 
Requirements,  Specific 

3.4  Performance 
Criteria,  Specific 

3.5  Measurement  Procedures 

4.  Planning  Components 
of  the  Process 

4.1  Analytic  Methods 

4.2  Parameter  Determinations 

4.3  Apparatus  for  Testing 

4.4  Personnel  for  Testing 

4.5  Test  Plans 

5.  Application  Components 
of  the  Process 

5.1  Test  Execution 

5.2  Data  Analysis 

5.3  Findings  interpretation 

5.4  Conclusions  and 
Recommendations 


Reliability,  availability,  and  maintain¬ 
ability  data  were  collected  on  the  product 
improved  items.  Personnel  skills  and 
training  requirements  were  also  identified. 

The  test  plan  utilized  was  the  USAARENBD 
Test  Design  Plan  for  Field  Evaluation  of 
M48A5  Tank  Product  Improvements  described 
in  TRADOC  Project  Number  1-VC-080-M48-602, 
May  1975. 

For  each  product  item,  detailed 
descriptions  were  presented  of  test 
procedures  including  objective,  method, 
analysis,  and  results. 


6.  Further  Research  Areas 

6.1  Measurement  System 
Limitations 

6.2  Research  Potentials/ 
Priorities 

6.3  Research  Planning 
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5.1  There  were  several  deviations  from  the  test 
plan.  These  included  no  firing  from  a 
moving  tank,  the  unavailability  of  the 
simulated  mission  firing  target,  and  the 
impossibility  of  accurately  scoring  gun 
engagements.  Conventional  silhouettes  were 
used  and  some  of  the  scoring  was 
subjective. 

5.3  It  was  concluded  that  the  product 

improvements  provided  the  system  with 
increased  capabilities  over  the  present 
system.  However,  several  safety  and  human 
factors  engineering  problems  should  be 
addressed  within  the  current  cost  and  time 
framework  of  the  development  of  the  system. 
Other  corrections  and  improvements  of  a 
minor  nature  were  recommended. 


Buckley,  E.P.,  Goldberg,  B.,  Rood,  R. ,  Hamilton,  H.  4  Champion,  P\ 
Development  of  a  performance  criterion  for  Enroute  Air  Traffic  Control 
personnel  research  through  air  traffic  control  simulation:  Experiment 
I-parallel  form  development  (FAA-RD-75-186,  Interim  Rep. ).  Atlantic  City, 
NJ:  Federal  Aviation  Administration,  National  Aviation  Ft  ilities 
Experimental  Center,  February  1976.  (AD-A023  411). 
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This  report  describes  one  of  a  series  of 
experiments  used  in  the  development  of  a 
standardized  cerformance  criterion  for 
journeyman  enroute  air  traffic  controllers. 
The  final  performance  measurement  system 
will  be  used  in  personnel  research  3uch  as 
the  evaluation  of  aptitude  tests  as  to 
their  capacity  to  predict  suitability  for 
entrance  into  training.  The  criterion 
measure  being  reported  here  will  be  based 
on  the  use  of  realistic  dynamic  simulation 
of  the  radar  air  traffic  control  situation. 
Its  specific  purpose  is  to  explore  one 
method  of  constructing  parallel  forms  of  a 
measurement  system. 

This  system  is  an  Enroute  Air  Traffic 
Control  system  consisting  of  the  National 
Aviation  Facilities  Experimental  Center 
(NAFEC)  dynamic  air  traffic  control 
simulator  and  two  controllers  working 
independently. 

Two  widely  divergent  sector  structures  were 
chosen  to  be  examined  with  three  traffic 
density  levels  each. 

Controllers  work  without  assistant 
controllers  and  communicate  with  "pilots" 
over  simulated  radio  frequencies. 

The  ultimate  performance  requirement  of  the 
Air  Traffic  Control  (ATC)  system  is  the 
safe  and  expeditious  movement  of  aircraft 
through  the  sector. 
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3.1  The  measurable  attributes  were  as  follows: 

Conf lictions,  delays,  completed  flights, 

communications,  identifications,  aircraft 

handled,  physical  effort  of  controllers. 

3.2  The  attribute  measures  for  this  experiment 

were  as  follows: 

(1)  Number  of  conflictions. 

(2)  Number  of  delays. 

(3)  Cumulative  delay  time. 

(4)  Number  of  completed  flights. 

(5)  Number  of  air/ground  contacts. 

(6)  Cumulative  air/ground  communication 
time . 

(7)  Number  of  aircraft  handled. 

(8)  Number  of  identifications  requested. 

(9)  Number  of  aircraft  in  sample. 

(10)  Number  of  completable  flights. 

(11)  Number  of  conflictions/number  of 
aircraft  handled. 

(12)  Number  of  conflictions/number  of 
delays. 

(13)  Number  of  delays/number  of  aircraft  in 
sample. 

(HD  Cumulative  delay  time/nuraber  of 
aircraft  in  sample. 

(15)  Number  of  completed  flights/number  of 
completable  flights. 

(16)  Number  of  contacts/number  of  aircraft 
handled. 
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(17)  Communication  tioe/number  of  contacts. 

(13)  Number  of  aircraft  handled/number  of 
aircraft  in  sample. 

(19)  Correlation  hold-delay  transformation. 

(20)  Number  of  identifications  requested 
minus  number  of  aircraft  in  sample. 

(21)  Controller  heart  rate. 

3.5  All  performance  measures  were  recorded,  and 
ratios  computed,  by  Mie  simulator  computer. 
Heart  rates  were  rec  ded  continuously 
throughout  the  experiment  and  compared  to  a 
resting  rate. 

4.1  Six  subjects  worked  the  traffic  control 
problem  in  each  of  two  sectors  at  three 
traffic  levels.  The  traffic  was  generated 
by  a  large-scale  digital  simulator  and 
directed  by  simulator  operators  who 
represented  pilots  in  the  real  ATC  system. 
The  computer  recorded  all  aircraft  events 
and  printed  the  performance  measure  scores 
at  the  end  of  one  hour. 

4.2  The  test  parameters  that  were  controlled 
were  as  follows: 

(1)  Sector  1  or  2. 

(2)  Traffic  density  of  40,  50,  60  aircraft 
per  hour . 

4.3  The  test  apparatus  was  the  National 
Aviation  Facilities  Experimental  Center 
dynamic  air  traffic  control  simulator. 

4.4  Six  qualified  enroute  air  traffic 
controllers  from  the  NAFEC  evaluation  group 
served  as  subjects. 
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5.2  The  basic  data  for  each  subject  were 
produced  in  histogram  form  for  each  of  the 
performance  measures.  A  three-factor 
analysis  of  variance  was  performed 
involving  the  variables  of  subjects  (6), 
sectors  (2),  and  traffic  densities  (3). 

5.3  The  results  indicated  that  the  hypothesis 
of  interaction  between  sector  and  density 
in  affecting  performance  was  not  sustained. 
There  was  little  difference  shown  in  the 
measures  between  sectors.  Great  difference 
was  shown  between  the  three  levels  of 
traffic  density. 

5.4  There  are  wide  differences  among  air 
traffic  controllers  in  their  ability  to 
handle  identical  traffic.  It  is  possible 
to  measure  these  differences  in  a 
completely  objective  manner.  The  sample  of 
subjects  was  small  and  the  data  points  few, 
thus  the  results  of  this  experiment  are 
only  indicative.  Future  experiments  must 
consider  the  problem  of  minimal  optimal 
traffic  sample  length;  one  hour  is  not 
enough. 
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Two  digital  computer  programs  synthesizing 
optimal  maneuvers  in  one-on-one 
air-to-air  combat  situations  are  described. 
The  method  develops  intelligently  interac¬ 
tive  maneuvers  without  relying  on  human 
pilot  experience.  One  program  drives  one 
of  the  interacting  aircraft,  thus  replacing 
one  of  the  human  pilots  on  the  NASA  Langley 
Research  Center's  Differential  Maneuvering 
Simulator,  this  in  real  time.  The  other 
program  operates  in  a  normal  batch  proc¬ 
essing  mode.  Both  programs  use  the  same 
technique  which  maps  the  physical  situation 
of  the  two  aircraft  into  a  quantized, 
abstract  situation  space.  The  outcome  in 
this  situation  space  is  predicted  for 
several  trial  maneuvers,  a  value  is 
associated  with  the  outcome  of  each  trial 
maneuver,  and  finally,  the  maneuver  with 
the  highest  predicted  value  is  executed. 

These  programs,  operating  with  six  degrees 
of  freedom  and  realistic  aerodynamic 
representation  for  both  aircraft,  provide  a 
means  for  objective  evaluation  of  weapons 
systems  and  pilot  performance. 


5.  Application  Components 
of  the  Process 

5.1  Test  Execution 

5.2  Data  Analysis 

5.3  Findings  Interpretation 

5.4  Conclusions  and 
Recommends  tl  ons 

6.  Further  Research  Areas 

6.1  Measurement  System 
Limitations 

6.2  Research  Potentials/ 
Priorities 

6.3  Research  Planning 
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In  discussing  the  effectiveness, 

reliability  or  readiness  of  a  weapon 

system,  the  following  items  must  be 

considered: 

(1)  General  characteristics  of  the  system. 

(2)  Operational,  tactical  and  strategic 
situation  for  which  the  system  is 
envisioned:  its  missions. 

(3)  Importance  of  the  system  relative  to 
other  systems. 

(4)  The  effectiveness  of  the  system 
against  various  target  types,  under 
various  operational  situations. 

(5)  Opportunity  for  using  the  system. 

(6)  Comparative  effectiveness  with  other 
existing  or  possible  competitive 
systems. 

(7)  Cost. 

(8)  Suitability  of  system  for  use  on 
ships,  planes,  etc. 

(9)  Types  of  ships,  planes,  etc.  on  which 
the  system  must  be  used. 

(10)  Ease  of  operation  and  maintenance. 

(11)  Reliability  and  operability. 


Further  Research  Areas 

6.1  Measurement  System 
Limitations 

6.2  Research  Potentials/ 
Priorities 

6.3  Research  Planning 
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(12)  Susceptibility  to  countei*taeasures. 


(13)  Susceptibility  to  interference  by  or 
with  other  systens  on  the  same  0 
neighboring  vehicle. 

(14)  Mobility  and  flexibility. 

3.1  Analysis  of  potential  effectiveness  of  a 

typical  weapon  system  is  concerned  with 

such  factors  as: 

(1)  Ability  to  detect  target. 

(2)  Ability  to  locate  and  identify  target. 

(3)  Ability  to  designate  target. 

(4)  Ability  to  track  target  for  fire 
control  purposes. 

(5)  Ability  to  bring  the  vehicle  bearing 
the  weapon  into  the  neighborhood  of 
the  target  soon  enough  to  permit  use 
of  the  weapon. 

(6)  Ability  to  bring  the  weapon  to  bear 
against  the  target  soon  enough  to  be 
effective,  once  in  range. 

(7)  Ability  of  weapon  system  to  place  the 
missile  within  the  desired  damaging 
radius  of  the  target. 

(8)  Ability  to  detonate  the  warhead  at  the 
proper  place,  in  the  proper  manner  at 
the  proper  time. 

(9)  Ability  of  warhead  to  inflict  the 
quality  of  damage  desired. 

(10)  Ability  to  fire  repeatedly  with  the 
necessary  degree  o'*  rapidity. 

(11)  Number  of  targets  that  may  be  engaged 
simultaneously  within  a  given  interval 
of  time. 
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3.2  To  each  of  these  factors  one  can  define 
suitable  measures  of  merit.  In  general 
each  such  factor  must  be  taken  into  account 
in  estimating  or  evaluating  the  worth  of  a 
weapon  system. 

One  of  the  common  measues  of  worth  of  a 
weapon  system  is  the  probability  that  one 
burst  of  a  missile  will  inflict  "Kill." 
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This  report  describes  the  efforts  and 
results  of  the  application  of  a  Systems 
Approach  to  Training  (SAT). 

The  system  addressed  was  the  A-6E  Pilot  and 
Bombardier  (B/N)  training  program.  The 
A-6E  tram  aircraft  is  a  two  man-subsonic, 
mid-wing  attack  aircraft.  It  is  manned  by 
a  pilot  and  borabardier/navigator . 

The  mission  of  the  system  is  to  perform 
high  and  low  altitude  all-weather  attacks. 
It  can  provide  close  air  support  for  ground 
forces  or  can  conduct  long  or  short  range 
interdiction  raids.  It  is  capable  of 
delivering  a  large  selection  of 
conventional  and  nuclear  weapons. 

The  analysis  encompassed  the  identification 
of  all  the  pilot  and  bombardier/navigator 
job  performance  requirements.  In  all,  over 
700  tasks  were  identified.  Task 
criticality,  frequency  of  occurrence, 
inherent  difficulty,  changes  in  knowledge 
and  skills  required  were  also  identified. 

A  Taxonomy  of  Training  Objectives  was 
specifically  developed  for  this  program. 
Criterion  objectives  were  determined  and 
criterion  referenced  tests  were  developed. 


5.  Application  Components 
of  the  Process 

5.1  Test  Execution 

5.2  Data  Analysis 

5.3  Findings  Interpretation 

5.4  Conclusions  and 
Recommendations 


6.  Further  Research  Areas 

6.1  Measurement  System 
Limitations 

6.2  Research  Potentials/ 
Priorities 

6.3  Research  Planning 
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1.3  This  report  summarizes  design-influencing 
human  factors  engineering  requirements 
applied  to  the  development  of  the  STINGER 
weapon  system.  The  system,  its  use,  and 
operating  procedures  are  provided  as 
background  for  the  human  factors 
engineering  components  of  program  planning, 
analysis,  design,  and  test  parts  of  the 
program. 

3.1  The  following  characteristics  of  the  total 
system  were  discussed  and  operationalized: 

(1)  Program  Planning 

(2)  General  System  Description 

(3)  Function  Allocation 

(4)  Critical  Tasks 

(5)  Weapon  System  Weight  and  Size 

(6)  Launch-Induced  Environment 


4.  Planning  Components 
of  the  Process 

4.1  Analytic  Methods 

4.2  Parameter  Determinations 

4.3  Apparatus  for  Testing 

4.4  Personnel  for  Testing 

4.5  Test  Plans 


(7)  Launcher  Controls 

(8)  Human  Engineering  Handling  Tests 

(9)  System-Handling  Performance 


5.  Application  Components  (10)  Training  Equipment 

of  the  Process 

5.1  Test  Execution 

5.2  Data  Analysis 

5.3  Findings  Interpretation 

5.4  Conclusions  and 
Recommendations 


6.  Further  Research  Areas 

6.1  Measurement  System 
Limitations 

6.2  Research  Potentials/ 
Priorities 

6.3  Research  Planning 
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Under  program  planning,  the  established 
goal  involved  the  design  and  development  of 
equipment,  facilities,  and  procedures  which 
would  provide  a  work  environment  conducive 
to  effective  work  patterns  and  personnel 
safety,  minimize  discomfort,  distraction, 
etc.,  and  downgrade  human  performance 
and/or  increase  error. 


A  second  part  of  the  program  planning 
involved  the  ”test  efforts”  with  an  eye 
toward  securing  data-relevant  selected  work 
cycles,  tests  in  which  human  participation 
is  critical  (speed,  accuracy),  use  of 
personnel  representative  of  the  military 
population,  collection  of  task  performance 
data,  identification  of  discrepancies 
between  required  and  obtained  tasks 
performance,  and  establishment  of  criteria 
for  acceptable  performance  of  tests. 

(Further  sections  of  this  document 
concentrate  on  narrow,  technical  aspects  of 
the  STINGER  weapon  system,  and  are  not 
relevant  to  the  objectives  of  this  review. 
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5.1  Test  Execution 
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5.4  Conclusions  and 
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1.1  Effective  quality  auditing  requires 
consideration  of  human  factors  at  each 
phase  of  the  process  from  initial  concept 
and  engineering  design  through 
manufacturing,  inspection  and  testing 
operations.  The  purpose  of  this  paper  is 
to  describe  human  factors  techniques  used 
to  obtain  better  understanding  of  basic 
quality  problems. 

3.2  The  first  requirement  for  improving 
inspection  accuracy  was  development  of  a 
standard  procedure  for  measuring  inspection 
effectiveness.  Job  sample  performance 
tests  were  developed  by  selecting 
representative  hardware  items  with  a  number 
of  known  defects.  These  measurements 
provided  basic  data  for  pinpointing  problem 
areas  and  evaluating  potential 
effectiveness  of  various  methods  for 
improving  inspection  accuracy. 

5.M  Research  has  indicated  that  low  inspection 
accuracy  may  be  due  to  a  number  of  specific 
factors,  such  as: 

(1)  Product  factors. 

(a)  Equipment  complexity 

(b)  Defect  rate 

(2)  Job  (inspection)  factors. 

(a)  Procedures 

(b)  Tools 

(c)  Visuals 
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(3)  Human  factors. 

(a)  Selection  of  inspection  personnel 

(b)  Training  of  inspection  personnel 


(c)  Motivation  of  inspection  personnel 
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1.  State  of  the  Art  Review 

of  the  Process 

1.1  General  System 
Measurements 

1.2  System  Taxonomy 
Model  (STM) 

1.3  Overall  Conceptual 
Process  Model  (CPM) 

2.  Contextual  Components 

of  the  Process 

2.1  System  Definition 

2.2  Mission  Definition 

2.3  Environment  Definition 

2.4  General  Constraints 
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1.1  As  shown  by  content  analyses  of  symposia 
papers  and  journal  articles,  significant 
differences  exist  between  tl  ?  related 
disciplines  of  "Ergonomics"  (European)  and 
"Human  Factors  Engineering"  (American). 
Ergonomics  appears  to  be  more 
physiologically — oriented.  Human  Factors 
more  psychologically-focused.  In  America 
there  has  been  more  concern  with 
integration  of  man  into  large  machine 
systems.  In  Europe  there  has  been  more 
concern  with  the  welfare  of  the  individual 
worker . 

1.3  The  difference  in  orientation,  in  essence, 
is  a  methodological  problem.  Ergonomists 
(and  human  factors  engineers)  are  more 
concerned  with  methodological  problems  than 
are  physical  scientists  and  engineers, 
because  precise  answers  about  the  behavior 
of  man  are  hard  to  find  in  any  handbook  or 
textbook.  Much  work  is  taken  up  with 
studies  in  which  methodology  is  of  such 
great  importance. 


4.  Planning  Components 
of  the  Process 

4.1  Analytic  Methods 

4.2  Parameter  Determinations 

4.3  Apparatus  for  Testing 

4.4  Personnel  for  Testing 

4.5  Test  Plans 

5.  Application  Components 
of  the  Process 

5.1  Test  Execution 

5.2  Data  Analysis 

5.3  Findings  Interpretation 

5.4  Conclusions  and 
Recommendations 

6.  Further  Research  Areas 

6.1  Measurement  System 
Limitation) 

6.2  Research  Potentials/ 
Priorities 

6.3  Research  Planning 


The  methods,  techniques,  apparatus,  and 
variables  used  by  psychologists  tend  to  be 
different  from  those  used  by  physiologists. 
Trying  to  decide  which  experimental  methods 
are  appropriate  to  any  practical  problem  is 
a  very  complex  question. 

The  value  or  worth  of  a  system  is  normally 
judged  by  several  criteria,  not  necessarily 
all  compatible.  Criteria  vary  greatly  from 
system  to  system,  and  many  iriteria  are 
specific  to  particular  systems.  Typical 
man  machine  system  criteria  include: 
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(1)  Anticipated  system  lifetime. 

(2)  Appearance. 

(3)  Comfort. 

(4)  Convenience. 

(5)  Ease  of  operation. 

(6)  Familiarity. 

(7)  Initial  cost. 

(8)  Maintainability. 

(9)  Manpower  requirements. 

(10)  Operating  cost. 

(11)  Reliability. 

(12)  Safety. 

(13)  Training  requirements. 

3.1  Common  ergonomic  and  human  factors  research 
dependent  measures  used  to  assess  system 
performance  include: 

.  Accuracy 
.  Cardiovascular 
response 

.  Critical  flicker 
fusion 
.  EEG 

.  Energy  expenditure 
.  Muscle  tension 
.  Psychophysical 
thresholds 


.  Ratings  (of 
comfort , 

annoyance,  etc.) 
.  Reaction  time 
.  Respiratory 
responses 
.  Spare  mental 
capacity 
.  Speed 

.  Trials  to  learn 


A-46 


i  Topics  Relevant 

I  to  System  Development 

1  | 
i Topi c} 

land  Evaluation  Technology 

!  No.  ! 

ABSTRACT 

5.3  One  of  the  most  important  methodological 
questions  with  which  one  has  to  come  to 
grips  is  how  can  measures  like  these  be 
matched  to  the  system  criteria?  One  needs 
to  concentrate  on  finding  combinations  of 
experimental  variables,  and  proper  weights 
to  assign  to  them,  to  arrive  at  an  overall 
index  of  what  is  relevant  and  important. 
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This  report  addresses  the  evaluation  of  a 
radar  imagery  recorder. 

The  equipment  evaluated  was  an  adverse 
weather  aerial  delivery  system  radar 
imagery  recorder  installed  in  a  C-130 
aircraft. 
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2.2  The  primary  purpose  of  the  evaluation  was 
to  determine  the  operational  effectiveness 
and  suitability  of  the  prototype  recorder 
system  for  use  in  navigator  ground 
training,  radar  prediction,  and 
reconnaissance/intelligence  gathering. 


3.  Analytic  Components 
of  the  Process 

3.1  Practical  Measurable 
Attributes 

3.2  Practical  Attribute 
Measures 

3.3  Performance 
Requirements,  Specific 

3.  *<  Performance 

Criteria,  Specific 
3.5  Measurement  Procedures 

0.  Planning  Components 
of  the  Process 

4.1  Analytic  Methods 

A. 2  Parameter  Determinations 
A. 3  Apparatus  for  Testing 
A. A  Personnel  for  Testing 
A. 5  Test  Plans 

5-  Application  Components 
of  the  Process 

5.1  Test  Execution 

5.2  Data  Analysis 

5.3  Findings  Interpretation 
5. A  Conclusions  and 

Recommendati ons 

6.  Further  Research  Areas 

6.1  Measurement  System 
Limitations 

6.2  Research  Potentials/ 
Priorities 

6.3  Research  Planning 


Specific  objectives  were  to  determine 
whether  the  system  provides  imagery  of  such 
quality  that  radar  returns  are  readily 
identifiable,  whether  the  system  can  be 
installed  in  the  aircraft  without  hindering 
operations,  and  the  operational  suitability 
of  the  navigator-operated  controls. 
Additional  objectives  were  to  determine 
whether  the  system  can  record  all  range 
marks,  leading  marks,  and  cursors;  whether 
the  system  is  capable  of  operating  at 
altitudes  up  to  and  including  25,000  feet 
above  mean  sea  level  with  the  aircraft 
pressurized  and  unpressurized. 

An  aircraft  simulator  was  designed  and  used 
by  the  engineers  to  exercise  the  prototype 
system.  The  system  was  installed  in  a 
field  training  detachment  Adverse  Weather 
Aerial  Delivery  System  (AWADS)  simulator  to 
check  compatibility  with  the  AWADS.  The 
C-130  AWADS  equipped  aircraft  was  used  as  a 
test  bed  for  the  prototype  system. 
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4.4  Ten  navigators,  three  of  whom  were 
qualified  weapons  and  tactics  officers, 
participated  in  the  study. 

4.5  Six  missions  of  eleven  sorties  were  flown 
under  controlled  test  conditions;  known 
checkpoints  and  offset  aiming  points  were 
used  by  the  navigators  to  provide 
independent  evaluation  of  the  system.  Each 
sortie  was  flown  at  preselected  altitudes 
ranging  from  500  feet  Above  Ground  L^-vel 
(AGL)  to  25,000  feet  Mean  Sea  Level  (MSL). 
Routes  were  preselected  and  enroute 
position  coordinates  were  recorded  on  data 
forms  along  with  intensity/gain  used.  A 
camera/periscope  assembly  recorded  the 
radar  display  and  auxiliary  data  throughout 
the  mission.  Debriefing  meetings,  attended 
by  various  specialists,  included  review  and 
analysis  of  the  recorded  imagery;  comments 
and  recommendations  were  solicited  from  the 
attendee  relative  to  his  area  of  expertise. 
Questionnaires  were  also  completed  by  the 
navigators  who  participated  in  the  study. 

5.4  It  was  concluded  that  the  prototype  system 
is  operationally  effective  and  suitable  for 
use  in  navigator  training  and  radar 
prediction  and  has  limited  capability  as  a 
reconnaissance/intelligence  gathering 
device.  Specific  recommendations  for 
improvement  of  the  system  were  made. 
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3.  A  Performance  _  _ 

Criteria,  Specific  3.2 

3.5  Measurement  Procedures 

A.  Planning  Components 

of  the  Process 
A.1  Analytic  Methods 
A. 2  Parameter  Determinations 
A. 3  Apparatus  for  Testing 
A. A  Personnel  for  Testing 
A. 5  Test  Plans 

5.  Application  Components 

of  the  Process 

5.1  Test  Execution 

5.2  Data  Analysis 

5.3  Findings  Interpretation 
5. A  Conclusions  and 

Recommendations 


System  effectiveness  is  based  on  a 
quantitative  measure  of  the  extent  to  which 
the  system  is  expected  to  meet  its  assigned 
role  in  a  specific  mission.  The  measure  is 
dependent  upon  system  parameters  of 
availability,  dependability  and  capability. 
The  capability  parameter  measure  the 
ability  of  a  system  to  achieve  specific 
mission  objectives,  given  that  the  system 
is  in  a  particular  operating  condition. 

A  system's  required  overall  capability  is 
directly  related  to  its  set  of  defined 
mission  objectives. 

System  capability  is  a  focal  parameter  in 
that  it  is  the  top  performance  parameter  of 
a  system  against  which  all  other  parameters 
are  funneled,  evaluated,  cross-traded  and 
optimized.  It  provides  the  linkup  of 
system  performance  with  mission  objectives. 

The  most  practical  and  realistic  frame  of 
reference  for  categorization  of  capability 
measures  is  by  specific  and  discrete  types 
of  major  missions  assigned  to  Air  Force 
squadrons,  wings,  or  unit  equipment.  A 
logical  initial  refinement  of  that  broad 
categorization  is  to  drop  down  and  recast 
the  measures  by  force  type.  The  final 
refinement  is  to  group  the  forces  by  common 
force  missions  and  stratify  the  resultant 
force  missions  by  discrete  types  of  major 
missions. 


6.  Further  Research  Areas 

6.1  Measurement  System 
Limitations 

6.2  Research  Potentials/ 
Priorities 

6.3  Research  Planning 
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The  report  presents  a  compendium  of 
capability  measures  for  each  of  the 
different  types  of  stratified  force 
missions.  F >urteen  (14)  stratified 
missions  are  listed,  for  which  a  total  of 
forty  (40)  capability  measures  are  defined. 


Because  overall  performance  response  of  the 
system  with  time  will  inherently  fluctuate, 
the  capability  measures  require  use  of 
statistical  methods  based  on  probability 
distribution  laws  or  use  of  empirical 
methods  for  their  evaluation. 


i.  state  of  the  Art  Review  1.1  This  paper  describes,  in  five  basic  points, 

of  the  Process  the  normative  technique  approach  to  the 

1.1  General  System  study  of  organizations. 

Measurements 


1.2  System  Taxonomy 

Model  (STM)  (1) 

1.3  Overall  Conceptual 
Process  Model  (CPM) 

2.  Contextual  Components 
of  the  Process 

2.1  System  Definition 

2.2  Mission  Definition 

2.3  Environment  Definition  (2) 

2.  <4  General  Constraints 

2.5  Performance 
Requirements,  Ultimate 

2.6  Performance 
Criteria,  Ultimate 

3.  Analytic  Components  .  . 

of  the  Process 

3.1  Practical  Measurable 
Attributes 

3.2  Practical  Attribute 
Measures 

3.3  Performance 
Requirements,  Specific 

3. K  Performance 

Criteria,  Specific 

3.5  Measurement  Procedures 

4.  Planning  Components 
of  the  Process 

44.1  Analytic  Methods 

41.2  Parameter  Determinations 
4(.3  Apparatus  for  Testing 

44. l|  Personnel  for  Testing 

41.5  Test  Plans 

5.  Application  Components 
of  the  Process 

5.1  Test  Execution 

5.2  Data  Analysis 

5.3  Findings  Interpretation  (5) 

5.44  Conclusions  and 

Recommendations 

6.  Further  Research  Areas 

6.1  Measurement  System 
Limitations 

6.2  Research  Potentials/ 

Priorities 

6.3  Research  Planning 


Organizations  are  goal  oriented  and 
the  goal  structure  can  be  translated 
into  a  "measure  of  performance"  such 
as  profitability,  benefit  minus  cost, 
social  utility,  etc. 

Organizations  can  be  subdivided  into 
components  which  themselves  have 
sub-goals.  However,  these  sub-goals 
must  necessarily  be  in  partial 
conflict. 

In  order  to  be  feasible,  it  is 
necessary  to  set  boundaries  of  the 
system  so  that  analysis  can  proceed  in 
an  orderly  fashion.  The  boundaries 
are  set  by  identifying  a  decision 
maker. 

The  systems  analyst's  task  is  to 
identify  one  or  more  important 
problems  of  the  decision  maker  and  to 
formulate  the  problems  so  that  they 
can  be  expressed  in  terms  of  a  model. 
The  model  must  be  rich  enough  to  lay 
out  the  alternatives  available  to  the 
decision  maker  and  to  enable  the 
systems  analyst  to  estimate  optional 
solutions. 

The  systems  analyst  should  have  an 
active  role  in  implementing  his 
"solutions." 


ABSTRACT 


A  system  is  defined  as  that  which  a 
decision  maker  can  control  and  change. 

The  environment  of  a  system  is  the  set  of 
things  which  the  decision  maker  cannot 
control  but  which  nevertheless  affect  the 
performance  of  the  system. 


( USAC-RP AO-005 ) .  Reading.  PA:  City  of  Reading, 
September  1971.  TPB-208  500-2). 
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1.  State  of  the  Art  Review 
of  the  Process 
1 . 1  General  System 
Measurements 
1.?  System  Taxonomy 
Model  (STM) 

1.3  Overall  Conceptual 
Process  Model  (CPM) 


1.1  This  volume  of  the  study  describes  the 

system  methodology  employed  by  the  City  of 
Reading,  Pennsylvania  for  urban  information 
systems. 

2.5  The  objectives  of  the  system  are: 


2.  Contextual  Components 
of  the  Process 

2.1  System  Definition 

2.2  Mission  Definition 

2.3  Environment  Definition 

2.4  General  Constraints 

2.5  Performance 
Requirements,  Ultimate 

2.6  Performance 
Criteria,  Ultimate 


(1)  A  system  design  that  is  transferable 
to  other  Municipalities. 

(2)  Production  of  fact  and  methodology  for 
rationalizing  the  information  flow  of 
the  Municipality. 

(3)  Definition  of  horizontal  loops. 


3.  Analytic  Components 
of  the  Process 

3-1  Practical  Measurable 

Attributes 

3.2  Practical  Attribute  .  * 

Measures  '5) 

3-3  Performance 

Requirements,  Specific  (5) 

3.4  Performance 
Criteria,  Specific 

3.5  Measurement  Procedures 

(7) 

4.  Planning  Components 
of  the  Process 

4.1  Analytic  Methods 

4.2  Parameter  Determinations  (8) 

4.3  Apparatus  for  Testing 

4.4  Personnel  for  Testing 

4.5  Test  Plans 

(9) 

5.  Application  Components 
of  the  Process 

5.1  Test  Execution 

5.2  Data  Analysis 

5.3  Findings  Interpretation 

5.4  Conclusions  and 
Recommendations 


Developing  a  regional  environment  for 
sharing  system  operation. 

Laying  groundwork  for  improvement. 

Consideration  of  future  linking  with 
the  state-wide  information  system. 

Utilization  of  other  on-going  Federal 
projects. 

Taking  advantage  of  existing 
information  system  technology. 

Viewing  the  city  as  a  "natural 
information  system"  with  inputs, 
processes,  and  outputs. 


6.  Further  Research  Areas 

6.1  Measurement  System 
Limitations 

6.2  Research  Potentials/ 
Priorities 

6.3  Research  Planning 
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(10)  Scheduling  to  utilize  project  funds 
appropriately. 

3.5  The  system  analysis  process  was  divided 
into  eight  phases: 

(1)  Organization  of  project. 

(2)  Research  -  literature,  field 
trip/technology,  studies,  workshops. 

(3)  Survey  and  data  collection. 

(4)  Synthesis  and  coarse  analysis. 

(5)  Data  conversion. 

(6)  Detailed  analysis. 

(7)  Report  preparation. 

(8)  Reanalysis. 

4.1  The  methods  associated  with  the  above 
phases  used  the  following: 

(1)  Technology  studies. 

(2)  Policy  guidelines,  application 
inventories. 

(3)  Annual  reports  from  departments, 
summaries  of  codes  and  ordnances. 

(4)  Flow  charts  of  current  systems. 

(5)  Reading  analysis  technique/event 
matrices,  decision  flow  charts. 

(6)  Statistical  tables. 
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1.  State  of  the  Art  Review 
of  the  Process 

1 . 1  General  System 
Measurements 

1.2  System  Taxonomy 
Model  (STM) 

1.3  Overall  Conceptual 
iVocess  Model  (CPM) 


1.3  A  Unit  Performance  Assessment  Model  (UPAM) 
was  designed  to  evaluate  simulated  2-sided 
actions. 

2.1  The  systems  addressed  are  rifle  squads, 
rifle  platoons,  and  tank  platoons. 


2.  Contextual  Components 
of  the  Process 

2.1  System  Definition 

2.2  Mission  Definition 

2.3  Environment  Definition 

2.4  General  Constraints  2.5 

2.5  Performance 
Requirements,  Ultimate 

2.6  Performance 

Criteria,  Ultimate  3,1 

3.  Analytic  Components 
of  the  Process 

3.1  Practical  Measurable 
Attributes 

3.2  Practical  Attribute 
Measures 

3-3  Performance 

Requirements,  Specific 

3.4  Performance 
Criteria,  Specific 

3.5  Measurement  Procedures  3.** 

4.  Planning  Components 
of  the  Process 

4.1  Analytic  Methods 

4.2  Parameter  Determinations 

4.3  Apparatus  for  Testing 

4.4  Personnel  for  Testing 

4.5  Test  Plans 

5.  Application  Components 

of  the  Process  o  c 

5.1  Test  Execution 

5.2  Data  Analysis 

5.3  Findings  Interpretation 

5.4  Conclusions  and 
Recommendations 

6.  Further  Research  Areas 

6.1  Measurement  System 
Limitations 

6.2  Research  Potentials/ 
Priorities 

6.3  Research  Planning 


The  missions  are  selected  kinds  of 
engagements  (e.g.,  attacking,  defending) 
with  the  enemy  for  each  system. 

The  ultimate  requirement  is  to  destroy  the 
enemy's  ability  to  wage  war. 

The  effectiveness  measures  include 
capturing,  immobilizing  or  defending  an 
objective  in  a  given  time  and  at  a  given 
cost.  In  the  study  example,  20  cost  and 
achievement  measures  were  selected  by 
military  experts  who  rated  a  larger  number 
of  objective,  quantitative,  face-valid 
measures. 

Performance  criteria  are  established  by 
having  experts  rate  the  significance  of  the 
various  cost  and  achievement  measures, 
calculating  and  applying  weights  to  those 
measures,  and  combining  the  set  into  a 
single  performance  criterion.  An  appendix 
provides  a  step  by  step  procedure  for 
setting  these  criteria. 

Measurements  are  made  through  the  use  of 
simulated  engagements  and  expert  estimates. 
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4.1  Weighted  variables  are  developed  by  having 
experts  rank  each  measure,  and  performing  a 
statistical  regression  on  the  set  of 
rankings.  Another  multiple  regression 
procedure  is  used  to  combine  these  measures 
into  a  single  index  of  performance  for 
comparison  with  pre-set  criteria. 

An  appendix  (K)  provides  a  step  by  step 
procedure  for  calculating  actual 
achievement  and  cost  scores,  and  computing 
the  performance  index. 

6.1  To  deal  with  the  limitations  wi*  this  study, 
a  cross  validation  effort  is  recommended  to 
test  the  efficiency  of  the  regression 
equations  used  in  calculating  the  index  of 
performance.  Also  situational  exercises 
should  be  used  to  validate  the  UPAM,  and  to 
provide  practical  application  guidelines. 


Coburn,  R.  Human  engineering  guide  to  ship  system  development  (NELC 
Tech.  Doc.  278).  San  Diego,  CA:  Naval  Electronics  Laboratory  Center, 
Ootober  1973.  (AD-772  535). 
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1.  State  of  the  Art  Review 
of  the  Process 

1 . 1  General  System 
Measurements 

1.2  System  Taxonomy 
Model  (STM) 

1.3  Overall  Conceptual 
Process  Model  (CPM) 


1.1  This  document  was  prepared  to  assist  Navy 
and  contractor  personnel  in  planning, 
managing  and  carrying  out  human  engineering 
programs  to  support  development  of  ship 
systems.  Very  little  attention  is  devoted 
to  issues  concerning  measurement  of  system 
performance  or  effectiveness. 


2.  Contextual  Components 

of  the  Process 

2.1  System  Definition 

2.2  Mission  Definition 

2.3  Environment  Definition 

2.4  General  Constraints 

2.5  Performance 
Requirements,  Ultimate 

2.6  Performance 
Criteria,  Ultimate 

3.  Analytic  Components 

of  the  Process 

3.1  Practical  Measurable 
Attributes 

3.2  Practical  Attribute 
Measu-es 

3-3  Performance 

Requirements,  Specific 

3.4  Performance 
Criteria,  Specific 

3.5  Measurement  Procedures 


Human  engineering  services  and  end  products 
relating  to  assessment  of  system 
performance  include: 

(1)  Man-Machine  Concept  Analyses  — 
Prediction  of  man-related  aspects  of 
system  performance  for  candidate  or 
selected  system  configurations. 

(2)  Man-Machine  System  Design  — 
Establishment  of  performance 
specifications  which  set  bounds  on 
man-machine  system  performance  and 
define  what  the  system  must  do  in 
operational  terms. 


4.  Planning  Components 
of  the  Process 

4.1  Analytic  Methods 

4.2  Parameter  Determinations 

4.3  Apparatus  for  Testing 

4.4  Personnel  for  Testing 
A. 5  Test  Plans 


5.  Application  Components 
of  the  Process 

5.1  Test  Execution 

5.2  Data  Analysis 

5.3  Findings  Interpretation 

5.4  Conclusions  and 
Recocxsendations 


6.  Further  Research  Areas 

6.1  Measurement  System 
Limitations 

6.2  Research  Potentials/ 
Priorities 

6.3  Research  Planning 
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1.  State  of  the  Art  Review  1.1 

of  the  Process 

1  I  General  System 
Measurements 

1.2  System  Taxonomy 
Model  (STM) 

1.3  Overall  Conceptual 
Process  Mod  i  (CPM) 

2.  Contextual  exponents 
of  the  Process 

2.1  System  Definition 

2.2  Mission  Definition 

2.3  Environment  Definition 

2.4  General  Constraints 

2.5  Performance 
Requirements,  Ultimate 

2.6  Performance 
Criteria,  Ultimate 


In  selecting  or  devising  a  measurement,  it 
is  essential  to  decide  cr  determine  the 
purpose  of  the  measurement.  In  industry 
the  purpose  translates  to  decisions  that 
management  or  personnel  people  must  make. 

The  second  element  in  defining  measurement 
concerns  what  is  to  be  measured.  In  job 
performance  evaluation  there  are  several 
categories  of  tests  available,  each 
measuring  different  things: 

(1)  Natural  observation. 

(2)  Job  sample  tests. 


3.  Analytic  Components 

of  the  Process 

3.1  Practical  Measurable 
Attributes 

3.2  Practical  Attribute 
Measures 

3 . 3  Performance 
Requirements,  Specific 

3 . 4  Performance 
Criteria,  Specific 

3.5  Measurement  Procedures 

4.  Planning  Components 

of  the  Process 

4.1  Analytic  Methods 

4.2  Parameter  Determinations 

4.3  Apparatus  for  Testing 

4.4  Personnel  for  Testing 

4.5  Test  Plans 

5.  Application  Components 

of  the  Process 

5.1  Test  Execution 

5.2  Data  Analysis 

5.3  Findings  Interpretation 

5.4  Conclusions  and 
Recommends  tions 

6.  Further  Research  Areas 

6.1  Measurement  System 
Limitations 

6.2  Research  Potential^/ 
Priorities 

6.3  Research  Planning 


(3)  Analytic  tests. 

(4)  Indirect  testT. 

(5)  Rating  scales. 

Measurement  effectiveness  or  validity  is 
best  described  by  what  one  should  consider 
in  dealing  with  it.  These  considerations 
are  in  the  form  of  the  following  questions: 

(1)  Accuracy  -  What  are  the  tolerances  of 
the  emerging  numbers? 

(2)  Stability  -  If  one  retested  later,  how 
similar  would  the  measurement  numbers 
be  to  the  first  set? 

(3)  Pay-off  -  How  much  better  are  the 
decisions  reached  using  measurement 
than  those  reached  without  such 
information? 
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1.  State  of  the  Art  Review  1.1 

of  the  Process 

1.1  General  System 
Measurements 

1.2  System  Taxonomy 
Model  (STM) 

1.3  Overall  Conceptual 
Process  Model  (CPM) 

2.  Contextual  Components 
of  the  Process 

2.1  System  Definition 

2.2  Mission  Definition 

2.3  Environment  Definition 

2.  A  General  Constraints  1.2 

2.5  Performance 
Requirements,  Ultimate 

2.6  Performance 
Criteria,  Ultimate 

3.  Analytic  Components 
of  the  Process 

3.1  Practical  Measurable 
Attributes 

3.2  Practical  Attribute 
Measure.’ 

3.3  Performance 
Requirements,  Specific 

3.  A  Performance 

Criteria,  Specific 
3.5  Measurement  Procedures 

A.  Planning  Components 
of  the  Process 
A.1  Analytic  Methods 
A. 2  Parameter  Determinations 
A. 3  Apparatus  for  Testing 
A. A  Personnel  for  Testing 
A. 5  Test  Plans 

5.  Application  Components 
of  the  Process 

5.1  Test  Execution 

5.2  Data  Analysis 

5.3  Findings  Interpretation 
5. A  Conclusions  and 

Recommendations 


One  of  the  problems  encountered  in  the 
definition  of  a  task  can  be  linked  to  the 
problems  associated  with  the  system  level 
to  which  the  taxonomy  is  being  applied. 
Another  problem  frequently  encountered  in 
defining  a  task  is  deciding  who  should 
define  the  task,  the  investigator  or  the 
operator:  what  the  scientist  says  a  person 
i*>  doing  may  or  may  not  conform  to  what  the 
person  thinks  he  is  doing. 

Two  issues  often  ignored  when  developing  a 
task  taxonomy  are  (1)  a  set  of  criteria, 
i.e.,  rules  on  which  a  Judgment  can  be 
based  for  the  evaluation  of  how  well  a  task 
taxonomy  accomplishes  the  goals  underlying 
its  development  and  (2)  the  relation 
between  taxonomic  structure  and  empirical 
data,  i.e.,  laboratory  and  field  data.  An 
effective  task  taxonomy  should  include  the 
following  criteria: 

(1)  The  taxonomy  must  simplify  the 
description  of  tasks  in  the  system 
because  the  goal  of  any  taxonomic 
schema  is  to  make  the  subject  matter 
more  manageable. 

(2)  The  taxonomy  should  be  generalizable. 
If  the  taxonomy  is  system  specific, 
the  effort  necessary  to  develop  it 
might  outweigh  the  benefit  derived. 


6.  Further  Research  Areas 

6.1  Measurement  System 
Limitations 

6.2  Research  Potentials/ 
Priorities 

6.3  Research  Planning 
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Also,  generalizability  is  congruent 
with  the  necessary  assumption  that 
activities  have  some  common  basis. 

(3)  The  taxonomy  must  be  compatible  with 
the  terms  used  by  others.  Unless  the 
taxonomy  is  in  a  form  that  is 
meaningful  to  those  who  will  use  it, 
its  application  will  be  inefficient 
and  often  ignored. 

(4)  The  taxonomy  must  be  complete  and 
internally  consistent,  i.e.,  it  must 
deal  with  all  aspects  of  human 
performance  in  the  system  without 
logical  error. 

(5)  The  taxonomy  must  be  compatible  with 
the  theory  or  system  to  which  it  will 
be  applied. 

(6)  The  taxonomy  should  help  to  predict 
operator  performance.  This  is 
necessary  to  evaluate  and  compare 
performance  between  operators  on 
different  as  well  as  identical  tasks. 

(7)  The  taxonomy  must  have  some  utility, 
either  practical  or  theoretical. 

(8)  The  taxonomy  must  be  cost  effective. 

It  is  possible  that  in  many  situations 
the  time  and  money  required  to  develop 
and  implement  a  task  taxonomy  may  add 
to  the  overall  cost  of  the  system  and 
provide  little  increase  in  operating 
efficiency. 

(9)  The  taxonomy  must  provide  a  framework 
around  which  all  relevant  data  can  be 
integrated.  Without  this  the  taxonomy 
i3  merely  a  verbal  device  with  no  ties 
to  reality  and,  therefore,  no 
applicability. 
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The  purpose  of  this  study  was  to  determine 
whether  inexperienced  people  can  be  trained 
to  apply  Teichner' s  Theoretical  Task 
Concepts . 

This  research  was  intended  to  provide  a 
first  evaluation  of  the  reliability  and 
validity  of  a  task  analysis  performed  with 
respect  to  Teichner 's  Theoretical  Task 
Taxonomy  and  instructional  procedures  for 
training  individuals  to  perform  the 
analysis . 

Problems  performed  on  desk  and  pocket 
calculators  were  developed  so  as  to 
represent  theoretical  tasks.  Ten  subjects 
were  instructed  in  the  theoretical 
concepts,  and  were  then  provided  a  partial 
operational  analysis  of  the  task  problem. 
They  were  then  required  to  complete  the 
operational  task  analysis  and  to  transform 
it  into  a  theoretical  task  analysis. 

Using  the  built-in  operational  and 
theoretical  steps  as  references,  the 
validity  of  the  subject's  procedures  was 
evaluated  in  terms  of  how  closely  the 
analysis  agreed  with  the  references. 

It  appears  that,  with  very  little  training, 
people  can  comprehend  the  concepts  and  be 
at  least  as  proficient  in  the  theoretical 
analysis  as  they  are  at  describing  actual 
operations.  Considering  that,  and  the 
general  level  of  performance,  it  is 
concluded  that  the  practicality  of  the 
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approach  is  supported,  i.e.,  operational 
task  descriptions  or  task  analysis,  can  be 
translated  correctly  into  the  tasks  of  the 
theory  by  uinimally  trained  observers. 

6.2  It  is  suggested  that  this  approach  should 
be  extended  to  the  evaluation  of  more 
complex  tasks. 


Connelly,  E.H.,  Bourne,  F.J.,  Loental,  D.G.  4  Knoop,  P.A.  Computer-aided 
techniques  for  providing  operator  performance  measures  (AFHRL-TR-74-87) . 
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1.1  The  problem  was  to  develop  and  implement 
standardized  techniques  for  deriving  and 
validating  measures  of  operator 
performance.  Traditional  techniques  involve 
hand-selecting  measures  which  appear  to 
have  content  validity,  then  testing  the 
measures  against  other  validation  criteria 
using  operator  performance  data.  This 
usually  results  in  a  resource-consuming 
iterative  research  process  that  is  often 
unsuccessful,  because: 

(1)  It  is  never  known  at  the  onset  whether 
or  not  the  most  useful  measures  have 
been  overlooked. 
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(2)  The  number  and  potential  validity  of 
measures  investigated  are  limited  by 
and  vary  with  the  researcher's 
ingenuity  and  the  time  available  for 
the  study. 

(3)  The  research  process  and  all 
associated  manual  effort  must  be 
repeated  for  each  new  measurement 
task. 

The  approach  was  to  develop  and  implement 
computer-aided  techniques  for  deriving  and 
validating  operator  performance  measures. 

A  "universal”  set  of  potential  measures  was 
defined  which  possesses  characteristics 
encompassing  many  traditionally  selected 
measures.  The  set  also  inherently  contains 
a  myriad  of  other  measures  whose 
characteristics  render  them  reasonable 
candidates.  Vectors  were  then  identified 
which  constitute  generators  for  the  set  of 
measures  (i.e.,  the  vectors  span  the 
defined  measure  space) .  Computational 
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algorithms  were  developed  which  generate 
and  operate  on  the  constituent  vectors 
using  multiple  regression  techniques. 
Several  empirical  validation  methods  were 
developed  for  testing  candidate  measures 
thereby  generated.  All  techniques  were 
implemented  in  a  computer-aided  measurement 
processor  which:  (1)  accepts  sample 
performance  data  and  various  user  inputs, 
and  (2)  generates  and  tests  candidate 
measures,  computes  statistics  for  assessing 
their  validity  likelihood,  and  prints 
results  for  user  analysis. 

2.1  The  sample  system  used  for  measurement  was 
a  T-37B  aircraft  in  flight  maneuvers. 

2.2  Five  specific  flight  maneuvers  were  flown: 

(1)  Cloverleaf. 

(2)  Split  S. 

(3)  Lazy  8. 

(4)  Normal  Landing. 

(5)  Barrel  Roll. 

3.1  Measurable  attributes  for  the  flight 
maneuver  sample  were  pitch,  roll,  heading, 
maneuver  sector,  airspeed. 

3.2  The  measures  were  in  degrees  for  pitch, 
roll  and  heading. 

4.1  Regression  analysis  was  used  to  generate 
reference  functions  which  are  representa¬ 
tive  of  excellent  performance. 

4.3  The  test  apparatus  consisted  of  an 
instrumented  T-37B  aircraft. 

4.4  Test  subjects  were  instructor  pilots  who 
purposely  demonstrated  both  good  and  bad 
maneuver  performances. 


A-65 


ABSTRACT 


5.2  The  developed  measurement  processor  was 
successfully  implemented  on  a  Sigma  5 
computer.  Demonstrations  of  the  operation 
of  the  software  were  performed  using  a 
liuited  amount  of  pilot  performance  data 
recorded  on  a  T-37B  aircraft.  The 
processor  performed  necessary  data 
smoothing,  automatically  segmented  the 
flight  maneuvers  for  measurement,  and 
developed  criterion  functions  from  the 
skilled  operator  data  provided.  Actual 
generation  and  validation  of  measures  was 
not  demonstrable  due  to  nonavailability  of 
originally  anticipated  data.  However, 
correct  software  performance  of  all  parts 
of  the  processor  was  verified. 

5.4  The  theoretical  concepts  and  computational 
techniques  underlying  the  developed 
measurement  processor  are  unique  and  have 
great  potential  for  operator  performance 
measurement  research.  The  applied  concept 
of  developing  a  set  of  vectors  which  span  a 
conceived  measure  space  and  operating  on  it 
with  regression  techniques  to  generate 
candidate  measures  is  itself  suggestive  of 
a  new  and  extremely  powerful  measurement 
tool.  The  processor  operation  can  be 
largely  independent  of  user  intervention; 
however,  it  is  also  capable  of  accepting 
user  inputs  reflecting  his  knowledge  about 
specific  measurement  problems.  It 
represents  a  truly  interactive  research 
system  wherein  user  tasks  as  distinguished 
from  processor  tasks  are  logically  defined, 
and  the  outcomes  of  each  are  integrated. 

Evaluation  of  the  adequacy  of  the  spanned 
measure  set,  the  generating  vectors,  and 
the  computational  mechanics  for  generating 
and  testing  measures  could  not  be  performed 
as  originally  planned  due  to  non-technical 
problems  which  prevented  the  collection  of 
required  data.  This  was  extremely 
detrimental  to  the  study  because:  (1)  many 
of  the  techniques  could  not  even  receive 
preliminary  test  prior  to  their 
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incorporation  in  the  processor,  and  (2)  the 
contributions  made  by  this  study  to  the 
general  technology  can  only  be  suggested 

instead  of  exemplified. 

♦ 

6.1  Follow-up  research  should  include 

derivation  of  the  basis  of  the  defined 
measure  set  using  the  implemented  processor 
as  an  aid  to  empirical  studies.  This  is, 
in  essence,  the  real  crux  of  the  operator 
performance  measurement  problem. 


Connelly,  E.M.,  Bourne,  F.J.,  Loental,  D.G.,  Migliaccio,  J.S.,  Burchick,  D.A 
&  Knoop,  P.A.  Candidate  1-3*"  pilot  performance  measures  for  five  contact 
maneuvers  (AFHRL-TR-74-88) .  McLean,  VA:  Quest  Research  Corp.,  December  1974. 
(AD-A014  331). 
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2.1  The  purpose  of  this  study  was  to  develop 

candidate  T-37  pilot  performance  measures, 
for  ultimate  use  in  an  advanced  simulator. 
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Five  undergraduate  pilot  training  contact 
maneuvers  were  selected  for  this 
development:  lazy  8,  barrel  roll,  split  S, 
cloverleaf  and  landing. 

The  first  step  of  the  approach  was  to 
analyze  each  maneuver,  using  techniques 
from  function  and  task  analyses,  in  order 
to  identify  candidate  measures. 

Several  algorithmic  measures  were  defined 
which,  collectively,  support  performance 
assessment  over  all  maneuver  segments. 
Content  validity  was  then  assured,  and  the 
user  specified  measures  were  then  computed 
using  specially  developed  software. 
Empirical  testing  was  to  be  conducted  next 
to  establish  criterion-related  validity. 
However,  non-technical  problems  prevented 
the  data  collection  phase  from  being 
completed. 

Specific  measurement  formula  combines  the 
various  measure  types  such  as:  continuous 
difference  error  measures  (used  to  identify 
the  deviation  from  a  trajectory),  threshold 
error  measures  (used  to  identify  violation 
of  a  threshold) ,  differential  difference 
error  measures  (used  for  time  dependent 
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changes  during  a  maneuver),  discrete  task 
error  measures  (used  to  identify  the 
accomplishment  of  a  discrete  event),  sample 
error  measures  (used  in  relation  to  a 
series  of  measures  during  a  specific 
maneuver),  and  miscellaneous  error 
measures. 

Computer  programs  were  developed  to: 

(1)  Smooth,  print  out,  and  plot  data 
recorded  on-board  a  T-37B  aircraft. 

(2)  Automatically  detect  task  segment 
boundaries. 

(3)  Compute  criterion  functions  from 
skilled  performer's  data. 

(U)  Compute  measures  specified  at  run-time 
by  the  user. 

(5)  Perform  and  print  results  of  several 
empirical  validation  tests  of  the 
candidate  measures  for  subsequent 
researcher  analysis. 


Connelly,  E.M.,  Schuler,  A.R.  4  Knoop,  P.A.  Study  of  adaptive  mathematical 
models  for  deriving  automated  pilot  performance  measurement  techniques 
(AFHRL-TR-69-7,  vol.  1:  Model  Development) .  Falls  Church,  VA:  Mel par , 
October  1969.  (AD-704  597). 
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Present  methods  of  (trainee  performance) 
measurement  consist  almost  exclusively  of 
ratings  or  judgments  made  by  skilled  pilot 
instructors/examiners.  Automatic  perform¬ 
ance  measurement  requires  measures  and 
criteria  in  precise,  quantitative  terras  in 
lieu  of  the  more  qualitative  terms  familiar 
to  human  evaluators. 

The  approach  used  in  this  study  is  quite 
different  (from  most  performance 
measurement  research) .  Instead  of  first 
deriving  measures  and  criteria  explicity 
and  molding  them  into  an  evaluation  system, 
the  authors  try  first  to  derive  a  reliable 
performance  evaluation  system  and  then 
analyze  the  system  to  determine  performance 
measures  and  criteria  required  for  valid 
and  reliable  performance  measurement. 

The  evaluation  system  is  derived  by 
developing  and  using  adaptive  mathematical 
and  computer  models  operating  on 
representative  performance  data 
corresponding  to  known  skill  levels.  The 
models  automatically  adjust  themselves  and 
their  methods  of  data  analysis  until 
capable  of  independently  evaluating 
performance.  Development  of  this  system  of 
adaptive  models  is  the  subject  of  this 
report. 

Automated  performance  evaluation  is  basic 
to  the  concept  of  adaptive  training.  The 
problem  is  to  develop  valid  and  reliable 
pilot  performance  measures  and  criteria  for 
use  in  automatically  and  objectively 
evaluating  trainee  performance  (on  a  flight 
simulator) . 
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2.4  The  characteristics  of  the  adaptive 
modeling  problem  are  as  follows: 

(1)  The  requirement  is  to  simulate/auto¬ 
mate  the  results  or  effect  the  trainee 
performance  assessment  process,  not 
necessarily  to  faithfully  simulate  the 
way  in  which  a  human  would  execute  the 
process. 

(2)  The  performance  assessment  process,  as 
performed  by  a  humar,  is  not  suffi¬ 
ciently  well  defined  or  understood  to 
allow  its  direct  implementation  on  a 
computer. 

(3)  Information  does  exist  about  the  data 
required  by  a  human  in  order  for  him 
to  perform  the  process. 

(4)  Some  skilled  humans  perform  the 
process  very  well. 

2.5  The  approach  described  in  the  report 
involves  development  of  an  adaptively  - 
programmed  computer  model  which  performs 
the  following  functions  iteratively: 

(1)  The  program  examines  actual  data, 
consisting  of  pilot  performance  data 
on  some  flight  maneuver  or  task;  many 
sets  of  representative  data  are 
needed. 

(2)  The  program  hypothesizes/approxiraates 
the  required  "process-effect,"  i.e.,  a 
score  or  rating  for  the  pilot  whose 
data  has  just  been  examined;  this 
requires  a  method  of  predicting  a 
score  from  actual  flight  data.  The 
study  developed  three  computational 
methods  for  doing  this,  each  of  which 
attempts  to  predict  score  or  a  portion 
thereof  in  a  unique  way. 
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(3)  The  program  compares  its  own  derived 
score  with  the  "true”  pilot  score  or 
rating  that  had  been  produced  for  the 
set  of  data  by  skilled  human  judgment. 

(4)  As  necessary,  the  program  adapts 
itself  and  its  methods  of  data 
analysis  to  improve  its  previous 
approximation . 

(5)  As  long  as  the  program's  derived 
ratings  do  not  correlate  sufficiently 
closely  with  the  experts'  ratings,  the 
program  repeats  the  preceding  steps 
with  a  new  set  of  data  and  continues 
to  adapt  itself.  But,  once  a 
sufficiently  close  correlation  is 
achieved,  the  program  shifts  to  a 
"testing"  mode,  wherein  it  ceases 
adaptation  and  produces  scores  for  new 
sets  of  data  for  purposes  of 
evaluation. 

3.2  The  three  computational  methods  used  by  the 
program  to  produce  pilot  scores/ratings  are: 

(1)  The  state  transfer  technique. 

(2)  The  relative  technique. 

(3)  Absolute  techniques. 

State  transfer  technique  is  based  on  the 
assumption  that  performance  skill  may  be 
partly  predictable  by  examining  trends  in 
the  performance  data.  Relative  technique 
assunes  that  performance  skill  may  be 
partly  predictable  by  examining 
relationships  among  separate  variables  in 
the  performance  data.  Absolute  technique 
assunes  performance  skill  may  be  partly 
predictable  by  examining  specific 
characteristics  of  performance  as  compared 
with  some  absolute  reference-performance. 
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4.1  The  results  produced  by  each  of  the 

computational  methods  consist  of  components 
of  the  performance  evaluation.  These 
components  are  routed  through  an 
"adaptive-mix,"  program  which  computes  a 
single  composite  performance  score. 

5.4  A  preliminary  demonstration  and  evaluation 
of  this  adaptive  modeling  system  produced 
encouraging  results. 


Connolly,  E.  M.  4  Sloan,  N.  A.  Manned  system  design  using  operator  measures 
and  criteria.  In  A.  S.  Neal  4  R.  F.  Palasek  (Eds.),  Proceedings  of  the  Human 
Factors  Society  21st  Annual  Meeting.  San  Francisco,  OctoDer  17-20,  1977. 
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The  designer  of  a  man-machine  system 
typically  performs  his  design  task  with 
knowledge  of  system  objectives,  human 
factors  principles,  and  display  and  control 
requirements.  However,  it  is  the  human 
operator  who  adapts  his  control  rule  (his 
input/output  control  characteristics)  so 
that  the  overall  system  responses  satisfied 
(to  the  degree  possible)  his  pei  formance 
criteria.  The  performance  that  is  actually 
achieved  will  be  obtained  in  c"operation 
with  a  system  that  has  a  good  system  design 
-  and  in  spite  of  a  system  having  a  poor 
design. 

The  above  argument  suggests  that  the 
designer  should  have  available  as  a  design 
tool  a  means  for  estimating  the  operator's 
performance  criteria  and  his  control 
actions.  The  designer  would  like  to  know 
which  design  features  support  performance 
and  which  features  degrade  performance. 

The  distance  between  the  intial  and  final 
positions  is  approximately  30  miles  and  the 
time  allotted  for  the  transit  is  90 
minutes. 

The  equipment  used  in  the  present 
investigation  included  a  surface  ship 
bridge  console  system  and  a  CRT. 


5.  Application  Components 
of  the  Process 

5.1  Test  Execution 

5.2  Data  Analysis 

5.3  Findings  Interpretation 

5.4  Conclusions  and 
Recommendations 

6.  Further  Research  Areas 

6.1  Measurement  System 
Limitations 

6.2  Research  Potentials/ 
Priorities 

6.3  Research  Planning 


4.5  In  this  study,  subjects  acting  as  Officers 
of  the  Deck  (OOD)  controlled  a  simulated 
ship  in  a  simulated  environment.  Their 
task  was  to  divert  a  ship  transit  from  the 
initial  point  to  the  terminal  point  within 
a  pre-specified  time  interval  while 
avoiding  simulated  contacts  along  the  way. 
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Past  performance  measurement  of  manual 
control  systems  has  used  a  single  summary 
measure  to  indicate  performance  of  the  total 
control  problem.  While  summary  measures 
give  the  total  picture,  they  do  not  indicate 
problems  that  arise  during  the  control 
problems . 

If  performance  is  monitored  continuously 
throughout  the  control  problem,  each  control 
action,  discrete  or  continuous,  can  be 
individually  evaluated.  The  Continuous 
Performance  Measure  (CPM)  could  be  used  to 
increase  the  efficiency  of  experiments, 
training,  and  design  of  manual  control 
systems.  This  report  documented  research 
that  deals  with  development  of  CPM  for 
flight  control  systems. 

The  desired  CPM  will  provide  the  correct 
motion  indication  of  the  aircraft  at  each 
point  of  the  mission  segment,  thus  providing 
a  flight  standard  against  which  actual 
flight  performance  can  be  compared. 

Moreover,  motion  errors  can  be  evaluated  in 
terms  of  their  significance  with  respect  to 
the  summary  performance  measure  in  question. 

Optimal  control  theory  can  determine  eacu 
aircraft  state  in  the  mission  segment  and 
the  optimal  control  and  solution  trajec¬ 
tories.  The  term  "optimal  solution  tra¬ 
jectories"  is  equivalent  to  the  aircraft 
motion  trajectories  that  minimize  the 
performance  measure  selected  for  that 
particular  mission  segment.  The  term 
"aircraft  state"  comprises  those  state 
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variables  that  describe  the  values  for  all 
aircraft  variables  involving  position  and 
rotation,  including  velocity. 

Several  model  components  were  constructed  in 
order  to  determine  what  is  "optimal": 

(1)  A  representation  of  the  current  task 
objectives  ard  weights  to  determine 
their  importance 

(2)  A  system  representation  of  what  is  to 
be  controlled,  and 

(3)  A  representation  of  limiting  or 
constraining  factors  on  the  system  and 
its  controls. 

A  mission  can  be  segmented  so  that  flight 
variables  may  be  specified  for  each 
segment.  The  total  mission  can  be  regarded 
as  a  series  of  segments  where  the  end- 
flight  conditions  of  one  segment  are  the 
beginning  conditions  for  the  following 
segment.  A  Segmented  Mission  Model  can 
thus  be  constructed  if  mission  segment 
specifications  are  converted  to  a  summary 
measure,  including  flight  constraints. 

In  order  to  convert  mission  specifications 
into  a  summary  measure,  a  cost  index 
function,  or  "penalty  function,"  was 
constructed  which  identifies: 

(1)  Deviation  from  the  desired  end  state, 
and 


(2)  Variable  rates  of  change,  control 

actions,  and  deviations  from  reference 
trajectories  occurring  along  the 
solution  path. 

The  particular  mission,  in  fact,  identified 
the  parameter  of  an  aircraft  operating  in  a 
Close  Air  Support  Night  Attack  mode. 
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5. 4  This  look  into  CPM  showed  that  summary 
measures  developed  from  mission  segment 
specifications  can  be  converted  into 
performance  measures,  this  was  done  by 
using  optimal  control  theory  to  either 
linearize  the  aircraft  equations,  or 
solving  the  optimal  control  for  the 
non-linear  aircraft  equations. 


Function  Allocation  Methodology  (HEFAM)  (Research  Memo.  SRM  70-11).  San 
Diego,  CA:  Navy  Personnel  and  Training  Research  Laboratory,  October  1969. 
('AD-699  173). 
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1.1  This  report  describes  an  effort  to  develop 
a  workable  cost/effectiveness  methodology 
for  man/machine  function  allocation.  The 
project  consisted  primarily  of  a  literature 
review  and  interviews  with  numerous 
technical  personnel  working  in  the  areas  of 
hunan  factors  and  personnel  research. 

The  Human  Effectiveness  Function  Allocation 
Methodology  (HEFAM)  was  conceived  as  an 
automated  data  storage  and  processing 
system  to  be  used  many  times  during  the 
development  cycle  of  new  weapon  and  support 
systems . 


Analytic  Components 
of  the  Process 


The  research  on  HEFAM  was  directed  toward 
three  major  areas  of  system  development: 


3.1  Practical  Measurable 
Attributes 

3.2  Practical  Attribute 
Measures 


(1)  Initial  Data  Collection. 
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(2)  Effectiveness  Quantification. 

(a)  Development  of  conceptual  basis 
for  human  effectiveness 
quantification. 


Application  Components 
of  the  Process 

5.1  Test  Execution 

5.2  Data  Analysis 

5.3  Findings  Interpretation 
5. A  Conclusions  and 

Recommendations 

Further  Research  Areas 

6.1  Measurement  System 
Limitations 
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(b)  Development  of  formulae  for  human 
effectiveness  quantification. 

(c)  Development  of  preliminary 
methodology  for  human 
effectiveness  prediction. 
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(3)  Overall  Conceptualization  of  the  HEFAM 
System. 

(a)  Data  bank  system. 

(b)  HEFAM  -  user  interface. 

Cc)  Relationships  between  th  HEFAM 
and  other  data  banks. 


5.3  The  findings  of  the  study  are  as  follows: 

(1)  There  has  been  much  effort  devoted  to 
the  problem  of  improving  human 
reliability  and  collecting  human 
reliability  data,  however,  there  has 
been  very  little  conceptualization  of 
the  overall  problem  of  quantifying 
human  effectiveness,  especially  in  the 
area  of  human  effectiveness 
prediction . 

(2)  There  is  a  lack  of  development  of  the 
state-of-the-art  of  quantifying  human 
performance  effectiveness. 

(3)  Mathematical  models  for  HEFAM  were 
determined,  computer  storage  capacity 
estimated  and  data  collection  methods 
proposed. 

(*0  The  type  of  data  required  and  the 
manner  in  which  it  is  to  be  stored 
were  stated. 

5. 4  The  study's  conclusions  are  as  follows: 

(1)  Much  more  work  effort  and  time  will  be 
needed  to  develop  the  HEFAM  system. 

(2)  The  HEFAM  system  will  consist  of  a 
methodology,  computational  formulae, 
and  an  automated  data  processing 
system. 
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(3)  More  sources  of  data  are  needed  in 

order  to  provide  a  large  enough  sample 
of  human  performance  to  form  a  data 
base  for  effectiveness  predictions. 


(4)  Simpler  methods  of  data  collection 
than  the  Operational  Sequence  Diagram 
(OSD)  method  are  needed  in  order  to 
collect  human  performance  data  in  a 
timely  manner. 

(5)  Computational  and  predictive  formulae 
for  HEFAM  must  be  developed  further  in 
order  to  be  used  for  actual 
computation  or  prediction. 

The  following  recommendations  are  made: 

(1)  It  is  recommended  that  the  HEFAM 
system  be  developed  as  rapidly  as 
possible.  This  will  require  more 
effort  than  is  currently  being 
expended. 

(2)  The  future  development  of  HEFAM  should 
include:  the  further  development  of 
the  conceptual  bases  of  human 
effectiveness  quantification;  the 
further  development  of  the  HEFAM  data 
processing  system;  the  collection  and 
utilization  of  human  performance  data; 
the  development  of  better  data 
collection  methods;  the  further 
development  and  testing  of 
computational  and  prediction  formulae; 
and  the  development  of  the  HEFAM 
prediction  methodology. 


Cunningham,  J.B.  A  systems-resource  approach  for  evaluating  organizational 
effectiveness.  Human  Relations,  1978,  31(7).  631-656. 


Topics  Relevant 

1  1 

1  1 

to  System  Development 

iTopici 

and  Evaluation  Technology 

!  No.  : 

ABSTRACT 

2. 


3. 


4. 


State  of  the  Art  Review 
of  the  Process 

1.1  General  System 
Measurements 

1.2  System  Taxonomy 
Model  (STM) 

1.3  Overall  Conceptual 
Process  Model  (CPM) 

Contextual  Components 
of  the  Process 

2.1  System  Definition 

2.2  Mission  Definition 

2.3  Environment  Definition 

2.4  General  Constraints 

2.5  Performance 
Requirements,  Ultimate 

2.6  Performance 
Criteria,  Ultimate 


1.2  The  basic  systems-raodel,  which  is  applied 
by  the  author  to  evaluate  organizational 
effectiveness,  deals  primarily  with 
sub-system  interrelationships.  Basic  to 
the  systems  model  is  an  analysis  of 
environmental  inputs,  methods  by  which  the 
inputs  are  transformed  (throughputs),  and 
the  end-products  of  this  transformation 
(outputs) . 

2.1  Characteristics  of  the  systems-model  are 
physical  and  chemical  laws  that  are 
applicable  to  social  organizations  as 
follows: 
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Every  system  uses  energy  in  a  cyclical 
way:  the  environmental  product  or 
output  becomes  the  energy  source  for 
the  subsequent  activity  cycle. 

Systems  are  separated  from  their 
environments  by  boundaries;  since 
events  are  structured  in  a  systematic 
way  in  an  organization,  the  boundaries 
of  the  system  are  between  events. 

Equifinality  in  open  systems:  a  final 
or  specific  end  state  can  be  reached 
by  a  diversity  of  inputs  and  varying 
environmental  and  internal  activities. 


5.  Application  Components  (i|) 
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5.1  Test  Execution 
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Entropy:  in  nature,  all  organized 
systems  "wind  down"  or  move  toward 
disorganization  and/or  death  -  this  is 
the  second  law  of  thermodynamics;  in 
open  systems,  however,  negative 
entropy  allows  the  system  to 
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temporarily  circumvent  entropy  by 
importing  more  environmental  energy 
than  it  expends. 

(5)  Equilibrium,  or  dynamic  homeostasis: 
systems  adapt  to  change  and  attempt  to 
maintain  a  balance  in  their  status 
quo;  the  system  will  also  attempt  to 
acquire  a  margin  of  safety  in  inputs 
above  and  beyond  what  it  needs  for 
mere  survival. 

(6)  Feedback:  an  information  input  into 
the  system  resulting  from  previous 
outputs  and  their  effect  on  the 
system's  environment. 

2.2  The  primary  goal  of  the  application  of  the 
systems  model  was  to  apply  it  to  real-world 
modeling  of  organizational  effectiveness: 
specifically,  seven  local  organizations 
(city  and  county  governments)  in  southern 
California  were  evaluated. 

The  effectiveness  of  these  organizations 
was  studied  in  terms  of  several  factors: 

(1)  The  organization's  ability  to  respond 
to  its  external  environment. 

(2)  The  organization's  ability  to  utilize 
resources  in  producing  outputs  and 
maintenance/restoration  of  the  system. 

(3)  The  organization's  ability  to  bargain 
and  optimize  its  use  of  resources  in 
an  environment  with  multiple 
decision-makers,  each  with  different 
goals. 

3.3  With  regard  to  the  first  factor  (responding 
to  the  external  environment),  four 
different  computer  simulation  problems  were 
designed  -  -  an  airplane  crash,  an 
earthquake,  a  flood,  and  ordinary 
day-to-day  problems  -  -  which  provided  a 
means  to  assess  the  adequacy  of  various 
organizational  resources  for  coping  with 
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these  disasters  of  various  magnitudes 
(including  variations  in  population 
density,  residential  structures,  and 
knowledge-levels) . 

The  specific  requirement  of  the  first 
factor  evaluation  was  to  measure  the 
organization’s  ability  to  achieve  the 
highest  resource  allocation  for  various 
levels  of  damage. 

The  organization’s  response  to  its  internal 
environment  -  -  its  efficiency  and 
bargaining  power  were  measured  in 
experimental  simulations  of  each  of  the 
seven  local  government  organizations. 

In  terms  of  the  third  factor,  a  specific 
allocation  of  resources  was  needed  for  a 
given  organizational  resolution  of  a 
problem;  the  decision  to  allocate  X  was 
made  on  the  basis  of  a  probable  payoff  Y. 
Since  a  decision-maker  should  logically 
allocate  resources  to  problems  of  the 
highest  payoff  first,  each  problem  was 
measured  in  terms  of  its  demand  for  X 
amount  of  resources. 

3.4  The  actual  measurement  of  resource 
availability  for  each  environmental 
possibility  was  computed  by  dividing  the 
total  cost  of  damage  by  each  subsystem’s 
reserve  of  resources. 

This  factor  was  measured  by  the  following: 
each  problem  is  assessed  in  terms  of  time, 
location,  cost,  and  amount  of  information 
given  to  the  decision-makers.  Each  of 
three  organizational  resources  -  vehicles, 
equipment,  and  personnel  -  were  then  given 
a  value  or  a  cost  as  an  indication  of  its 
value  to  the  organization.  The  resource 
value  needed  for  the  resolution  of  a  given 
problem  could  then  be  calculated. 
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The  value  of  resources  theoretically 
necessary  to  respond  to  problem  j  is: 

50Pj  =  I  <pJk  •  rrk> 

P  ..  is  the  number  of  resources  of  type  k 
wnich  are  the  standard  operational  response 
to  problem  j. 

rr^  is  the  value  of  resource  k. 

SOP  is  the  Standard  Operational  Response. 

4.1  The  data  for  the  above  were  collected 
through: 

(1)  Use  of  a  panel  of  experts  to  predict 
problems  and  events  occurring  with  an 
organization's  external  environment. 

(2)  Prediction  of  possible  environments  by 
computer  simulations. 

(3)  Accounting  of  resource  availability 
for  each  local  governmental 
department. 

(4)  Use  of  a  gaming  simulation  experiment 
measuring  the  decision-maker's 
efficiency  and  bargaining  capability. 

5.2  The  decision-maker's  ability  to  correctly 
interpret  the  external  environment  was 
calculated  through  the  use  of  F  ratios 
indicating  the  degree  of  difference  among 
the  resource  ratios  (resource 
available/possible  damage  levels)  of  the 
various  organizational  subsystems.  These 
analytic  techniques  were  also  repeated  for 
factors  2  and  3  (organizational  efficiency 
and  bargaining  capability). 

5.3  Private  organizations  were  found  to  respond 
at  a  much  higher  level  than  the 
governmental  organizations  studied;  this, 


ABSTRACT 


however,  does  not  necessarily  indicate 
greater  efficiency:  private  organizations 
over-responded  as  a  public  relations 
gambit,  not  just  in  response  to  the 
"problem".  There  was  also  greater 
variation  among  private  organization 
efficiencies  than  there  were  in  the  public 
sector. 


Cunningham,  R.P.,  Sheldon,  M.S.  4  Zagorski,  H.J.  Project  NORM:  Pilot  study 
report  (TM-2232/000/00) .  Santa  Monica,  CA:  System  Development  Corp., 
February  1965.  I AD-458  341). 
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Historically,  there  has  been  much 
controversy  concerning  the  measures  used  to 
assess  performance,  some  of  which  purport 
to  evaluate  functional  units  of  the  system, 
while  others  deal  with  subsystems  and  still 
others  attempt  to  assess  the  behavior  of 
the  total  system.  Littl.*  is  known  about 
the  relationship  among  the  various 
measurements  or  their  relevance  as  criteria 
for  making  adequate  judgments  regarding 
training  operations.  Frequently,  it  has 
been  asserted  that  single  performance 
measures  are  inadequate  for  making  overall 
evaluations  of  system  effectiveness, 
presumably  because  multiple  factors  are 
involved  in  the  determination  of  mission 
success  or  failure.  Combining  measures 
into  overall  indices  has,  so  far,  not 
seemed  to  help  much,  probably  because  the 
relationship  between  them  is  not  clearly 
understood.  Thus,  lumping  them  together 
does  not  necessarily  improve  the  quality  of 
system  evaluation. 

Project  NORM  (Normative  Operations 
Reporting  Method)  was  a  pilot  study  for  a 
subsequent  comprehensive  research  effort 
aimed  at  obtaining  a  better  understanding 
of  air  defense  mission  evaluation  and 
analysis. 


5.  Application  Components 
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A  mission  problem  containing  37  fakers  was 
tested.  A  computer  printout  was  obtained 
and  data  relevant  to  41  situational, 
subsystem  and  system  performance  variables 
were  extracted.  Those  variables  actually 
considered  in  the  study  were: 
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(1)  Detection  latency. 

(2)  Total  track  off  time. 

(3)  Commitment  latency. 

(4)  Distance  interceptor  -  faker  at  first 
pairing. 

(5)  Faker  target  life. 

(6)  Faker  distance  travelled. 

(7)  Faker  distance  to  nearest  ground 
target. 

4.1  Statistical  treatment  of  the  data  was  made 
with  the  aim  of  demonstrating  how  prior 
knowledge  of  the  mission  situation  can  be 
used  to  predict  expected  performance  for 
the  purpose  of  gauging  crew  progress. 

This  pilot  study  clearly  demonstrated  the 
feasibility  of  addressing  the  criterion 
development  problem  in  a  quantitative  and 
scientific  fashion. 
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This  report  provides  guidance  on  test  and 
evaluation  (T&E)  at  two  distinct  levels. 

At  the  most  general  level,  this  report 
discusses  a  number  of  issues  which  are 
appropriate  for  all  weapon  system  acquisition 
programs.  In  addition,  a  general  checklist 
of  items  is  presented  which  is  organized  for 
a  rapid  overall  review  of  T&E  aspects, 
generally  applicable  to  all  systems  develop¬ 
ment  and  deployment.  A  sample  of  the  recom¬ 
mendations  is  included  in  this  abstract. 

Demonstration  and  acceptance  tests,  as  well 
as  tests  intended  to  evaluate  performance 
under  operational  conditions,  should  always 
be  conducted  under  conditions  as  close  to 
those  anticipated  in  practice  as  possible. 

On  the  other  hand,  test  conditions  during 
development  should  be  determined  by  the 
primary  objectives  of  that  test,  rather  than 
by  more  general  considerations  of  realism, 
etc.  Whenever  a  non-tactical ,  non-operational 
configuration  is  dictated  by  test  require¬ 
ments,  the  results  of  the  tests  should  not 
be  challenged  by  the  fact  that  that  config¬ 
uration  was  not  tactical  or  operational . 

By  the  end  of  the  systems  definition  phase, 
it  should  be  made  certain  that  test  criteria 
are  established  so  that  there  is  no  question 
as  to  what  constitutes  a  test  and  what  per¬ 
formance  is  to  be  obtained.  A  relationship 
between  the  identified  performance  parameters 
and  the  test  results  should  be  established 
prior  to  the  conduct  of  the  test.  Further, 
the  set  of  objectives  for  each  of  the  tests 
should  be  clearly  related  to  the  program 
objective. 
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Each  performance  characteristics  specified 
should  be  measurable  through  bench  and 
laboratory  or  proving  ground  testing.  The 
test  design  and  the  number  of  tests  should 
be  adequate  to  provide  results  with  con¬ 
fidence  limits  compatible  with  the  state¬ 
ments  of  desired  characteristics.  Testing 
in  advanced  development  should  be  planned 
to  explore  performance  characteristics  over 
a  broad  range  of  environments  so  as  to  pro¬ 
vide  insight  into  system  performance  over 
the  expected  operational  range  and  not  just 
at  a  single  point. 

3.5  A  good  test  program  makes  provisions  for 
feedback  of  test  results,  during  conduct 
of  the  testing,  so  as  to  influence: 

(1)  Course  of  the  T&E  program  (test 
director,  program  manager). 

(2)  Tradeoff  decisions  between  modifying 
the  system  design  and  relaxing  the 
operational  requirements  (program 
manager,  operating/supporting 
commands,  HQ). 

(3)  Missions,  employment  doctrine,  tactics 
and  constraints,  tactical  organization, 
etc.  (operating  command,  operational 
units ) . 

(4)  Parts  provisioning. 

When  developing,  testing  and  evaluating  the 
various  subsystems  (and  systems)  of  non¬ 
expendable  weapon  systems,  each  component  of 
the  systems  should  be  numbered  and  a  perfor¬ 
mance  history  kept  which  allows  an  analysis 
of  that  component’s  performance  with  respect 
to  reliability,  maintainability,  availability, 
etc.  An  analysis  of  failure  modes  should  be 
made  in  advance  so  as  to  relate  test  results 
to  the  operational  capability  of  the  system 
when  in  a  degraded  condition. 
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Techniques  and  system  range  instrumentation 
should  be  developed  tc  provide  the  type  of 
data  in  the  proper  form  to  allow  econimic, 
analytical,  and  mechanical  simulation  for 
alternate  scenarios  and  combinations. 

4.3  If  there  are  government  furnished  equipment 
(GFE)  and  other  government  commitments  in 
the  proposed  contract,  be  concerned  about 
the  following: 

(1)  Can  the  gear  with  required  performance 
be  available  when  required? 

(2)  Can  government  supported  facilities 
provide  the  assistance  required  at  the 
time  needed?  If  not,  is  it  reasonable 
to  construct  the  required  facilities 
(test  range,  instrumentation,  building, 
etc.)?  If  not,  what  alternatives  are 
available? 

Whenever  possible  the  initial  phase  of 
operational  test  and  evaluation  (IOT&E)  of 
a  weapon  system  should  be  planned  to  include 
other  systems  which  must  have  a  technical 
interface  with  the  new  system.  Thus,  missiles 
should  be  tested  on  most  of  the  platforms  for 
which  they  are  programmed.  Interfaces 
between  system  should  receive  special  attention 

The  manner  in  which  T&E  instrumentation  is 
used  can  be  extremely  important  in  determining 
the  realism  possible  in  the  OT&E  phases.  The 
instrumentation  package  should  be  fixed  early 
in  the  design  phase  of  the  development;  it  is 
difficult  and  costly  to  change  thereafter. 

For  this  reason,  instrumentation  require¬ 
ments  must  be  specified  early  in  the  program 
and  operational  factors  must  be  incorporated 
early. 

The  applicability  of  existing  test  ranges  and 
the  adequacy  of  facilities  and  instrumenta¬ 
tion  should  be  verified. 
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4.4  Testers,  evaluators  and  operators  have 
quite  different  backgrounds  and  needs 
which  affect  the  T&E  of  the  weapon  system. 

Each  has  a  different  approach  which  has 
merit  and  utility  at  almost  all  points  in 
the  T&E  program.  A  mix  of  these  types  is 
needed  throughout  the  program.  Early  in 
the  program,  the  lead  emphasis  should  be 
from  the  tester,  shifting  to  the  evaluator 
and  finally  the  operator,  but  at  all  times 
all  parties  and  their  needs  should  be 
coordinated . 

Training  plans  and  certification  plans  for 
test  personnel  should  be  established  early 
in  the  Full-Scale  Engineering  Development 
Phase.  Errors  by  test  personnel  are  usually 
expensive  and  often  cloud  the  reason  for 
test  failures. 

It  is  imperative  that  the  Independent  Test 
Agency  participate  in  all  of  the  T&E  phases 
to  ensure  that  the  user  needs  are  repre¬ 
sented  in  the  development  of  the  system 
concept  and  hardware.  Initially,  the  Inde¬ 
pendent  Test  Agency  should  play  an  advisor 
role  during  the  feasibility  and  engineering 
testing,  and  gradually  take  over  leadership 
in  the  conduct  of  the  testing  program  as  it 
becomes  more  and  more  operational.  This 
should  facilitate  the  necessary  communication 
and  interaction  between  developing  and  user 
commands . 

The  test  director  and/or  key  members  of  the 
test  planning  group  within  the  project  office 
should  have  significant  T&E  experience.  If 
the  requisite  experience  does  not  exist  at 
the  appropriate  levels  within  the  project 
office,  test  plans  may  be  based  on  too  shallow 
or  too  naive  a  conception  of  the  role  and 
potential  utility  of  the  T&E  process.  All 
too  often,  key  test  personnel  are  assigned 
to  T&E  slots  with  little  prior  exposure  to 
T&E  or  its  management,  and  with  inadequately 
experienced  support  as  well.  The  test  planning 
group  should  have  personnel  experienced  in 
engineering  testing,  deveDopment  testing  and 
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and  operational  testing.  This  experience 
should  be  available  very  early  and  all 
efforts  should  be  made  to  encourage  these 
people  to  remain  with  the  weapon  system 
project  office  through  the  T&E  phases  of 
the  program. 

The  planners  and  evaluators  for  the  OT&E 
of  the  production  equipment  can  do  a  better 
job  if  they  are  initially  involved  in 
planning  and  conduct  the  IOT&E. 

In  the  initial  conduct  of  OT&E,  the  partici¬ 
pants  should  be  given  a  period  of  time  to 
dry  run  the  scenarios  and  to  shake-down  the 
instrumentation  and  the  overall  operation 
before  key  resources  are  expended  in  tests 
for  record.  In  a  properly  planned  OT&E 
program,  the  people  will  have  completed 
proper  individual  training  on  the  new  sys¬ 
tem  but  the  operational  organization  will 
not  be  able  to  conduct  full  unit  training 
until  the  hardware,  software,  and  support 
equipment  are  on  hand.  After  the  period 
when  the  unit  is  qualified  as  being  opera¬ 
tionally  ready,  it  would  be  ready  for 
assignment  to  OT&E  testing. 

Test  conduct  can  be  influenced  by  the  actions 
of  the  observers  and  umpires.  These  people 
can  provide  important  clues  to  the  participants 
of  operational  suitability  testing  and  in  that 
way  lesson  the  validity  of  the  test.  For 
example,  in  situations  where  air/ground  duels 
are  to  be  conducted,  briefed  observers  who 
look  in  the  direction  of  the  aircraft,  might 
inadvertently  tip-off  the  direction  of  approach 
to  the  ground  party  in  the  duel.  Similarly, 
concentrations  of  observers  at  a  certain 
location  may  clue  the  aircrews  where  to  search 
first  for  the  ground  targets. 

l|.5  Every  test  plan  should  include  clear  state¬ 
ments  of: 

(1)  The  overall  purpose  of  the  test. 

(2)  Critical  issues  with  respect  to 
operational  requirements. 
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(3)  The  major  test  objectives. 


(iJ)  The  schedule  of  test  milestones. 

(5)  The  major  resources  required: 

(a)  Test  environment,  facilities  and 
instrumentation. 

(b)  Operational  environment 

(6)  The  organizations  which  will  conduct 
the  test  program. 

(7)  The  analysis  and  evaluation  approach. 
Tests  should : 

(1)  Have  specific  objectives. 

(2)  List  in  advance  actions  to  be  taken 
as  a  consequence  of  the  test  results. 

(3)  Be  instrumented  to  permit  diagnosis  of 
the  causes  of  lack  of  performance. 

(4)  Not  be  repeated  if  failures  occur, 
without  detailed  analysis  of  the  failure. 

(5)  Be  rehearsed  for  each  new  phase  of 
testing. 

The  test  schedule  should: 

(1)  Allow  for  a  sufficient  time  between  the 
planned  end  of  demonstration  testing 
and  major  procurement  decisions  so  that 
there  is  a  flexibility  for  modification 
of  plans  which  may  be  required  during 
the  test  phases  of  the  program: 

(a)  The  number  of  test  items  available 
and  the  schedule  interface  with 
other  systems  needed  in  the  tests, 
such  as  aircraft,  electronics, 
missiles,  etc. 

(b)  Support  required  to  assist  in  the 
preparation,  conduct  of  the  tests, 
and  the  analysis  of  t°st  results. 
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(2)  Be  adjusted  to  minimize  the  so-called 
T&E  gap  caused  by  a  lack  of  hardware. 
Specifically,  a  test  gap  can  result 
if  funds  are  not  applied  until  the 
results  of  IOT&E  are  known  because  of 
the  required  lead  time  for  production 
planning,  production  facilities,  and 
tool  and  production  hardware. 

Budgeting  documents  should  be  regularly 
reviewed  to  ensure  that  there  are  adequate 
identified  funds  for  testing,  relative  to 
development  and  fabrication  funds.  Budget¬ 
ing  documents  need  careful  scrutiny  to 
ensure  that  there  are  adequate  contingency 
funds  to  cover  correction  of  difficulties 
at  a  level  which  matches  the  Industry/ 
Government  experience  on  such  contracts. 
(Testing  for  difficulty  without  sufficient 
funding  for  proper  correction  results  in 
band  aid  approaches  which  ultimately  require 
correction  at  a  later  and  more  expensive 
time  period . ) 

The  constraints  to  be  placed  on  the  tes' 
because  of  the  range  and  instrumentation 
are  of  prime  importance.  As  previously 
stated,  the  test  facilities  and  instruments 
tion  requirements  to  conduct  operational 
tests  should  be  identified,  along  with  a 
tentative  schedule  of  test  activities.  The 
applicability  of  existing  test  ranges  and 
the  adequacy  of  current  facilities  and  in¬ 
strumentation  should  be  verified.  Insofar 
as  possible,  alternative  approaches  (differ 
ent  ranges,  etc.)  and  instrumentation 
improvements  needed  should  be  specified.  If 
range  and  instrumentation  factors  are  found 
to  cast  significant  doubt  on  the  meaningful 
ness  of  the  test  data  because  of  a  lack  of 
operational  realism,  the  steps  necessary  to 
assure  meaningful  data  should  be  previously 
identified  and  planned. 

The  primary  basis  for  the  test  sample  size 
is  usually  based  on  one  or  more  of  the 
following: 

(1)  Analysis  of  test  objectives. 
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(2)  Statistical  significance  of  test 
results  at  some  specified  confidence 
level . 

(3)  Availability  of  test  vehicles,  items, 
etc . 

(ij)  Support  resources  or  facilities 
available. 

(5)  Time  available  for  the  test  program. 
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1.  State  of  the  Art  Review 
of  the  Process 

1 . 1  General  System 
Measurements 

1.2  System  Taxonomy 
Model  (STM) 

1.3  Overall  Conceptual 
Process  Model  (CPM) 


1.1  The  purpose  of  this  paper  is  to  present  a 

descriptive  model  of  the  evaluative  process 
that  has  been  synthesized  from  the  concepts 
presented  in  numerous  papers.  In  addition 
emphasis  was  placed  on  the  criterion 
problem. 


2.  Contextual  Components 
of  the  Process 

2.1  System  Definition 

2.2  Mission  Definition 

2- 3  Environment  Definition 

2.4  General  Constraints 

2.5  Performance 
Requirements,  Ultimate 

2.6  Performance 
Criteria,  Ultimate 

3.  Analytic  Components 
of  the  Process 

3.1  Practical  Measurable 
Attributes 

3.2  Practical  Attribute 
Measures 

3- 3  Performance 

Requirements,  Specific 

3.4  Performance 
Criteria,  Specific 

3.5  Measurement  Procedures 

*1.  Planning  Components 
of  the  Process 

4.1  Analytic  Methods 

4.2  Parameter  Determinations 

4.3  Apparatus  for  Testing 
4  4  Personnel  for  Testing 

4.5  Test  Plans 

5.  Application  Components 
of  the  Process 

5.1  Test  Execution 

5.2  Data  Analysis 

5.3  Findings  Interpretation 

5.4  Conclusions  and 
Recommendations 


Evaluation  of  training  should  be  considered 
in  the  initial  developmental  stages  of 
program  design.  The  evaluation  process 
consists  primarily  of  translating  the 
program  objectives  into  quantifiable 
measures  which  can  be  used  to  feed  back 
into  the  program  to  indicate  needed 
modifications  or  as  the  final  information 
used  in  making  the  decision  as. to  whether 
the  program  should  continue  or  not.  The 
major  problem  in  training  evaluation  is  the 
translation  of  objectives  into  criteria. 

The  criterion  is  a  measurable  variable 
which  represents  the  dimension  of  interest 
in  the  research  study.  In  industrial 
psychology  the  criterion  is  usually 
predicted  from  knowledge  of  the  predictor 
variable  measure.  The  greater  the  ability 
of  the  predictor  variable  to  predict  or 
capture  the  criterion  variable  the  higher 
the  validity  of  the  predictors. 

Three  criterion  concepts  are  presented: 

(1)  Ultimate,  intermediate,  immediate. 

(2)  External,  internal. 


6.  Further  Research  Areas 

6.1  Measurement  System 
Limitations 

6.2  Research  Potentiala/ 
Priorities 

6.3  Research  Planning 
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(3)  Sunmative,  formative. 

Each  type  in  each  concept  is  defined  as 

follows: 

(1)  Ultimate  criteria  are  usually  stated 
in  brocd  conceptual  terms  and 
represent  the  total  goal  of  a 
particular  action. 

(2)  Immediate  criteria  are  some  variable 
measures  which  are  obtainable  and  are 
logically  felt  to  represent  the 
ultimate  criteria. 

(3)  Intermediate  criteria  are  some 
measurable  variables  that  are 
logically  felt  to  be  related  to  the 
Immediate  variables  and  which  are 
obtainable  prior  to  attaining  the 
immediate  criteria. 

(4)  Internal  criteria  are  considered 
outcome  measures  linked  directly  to 
the  training  content  which  are 
assessments  made  during  or  immediately 
after  the  learning  experience. 

(5)  External  criteria  are  designed  to 
assess  behavioral  changes  in  the 
organizational  role  of  the  individual. 

(6)  Formative  criteria  are  those  used  to 
evaluate  segments  of  the  training 
program  to  make  changes  within  the 
program  prior  to  its  culmination. 

(7)  Summative  criteria  are  concerned  with 
the  final  evaluation  of  the  total 
program. 

Three  dimensions  of  criteria  are  discussed: 

(1)  The  static  dimension  which  is  the 
usual  type  of  criteria  of  production 
or  Job  performance  behavior. 
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(2)  The  dynamic  dimension  which  is  the 
change  over  time  of  Job  performance. 

(3)  The  individual  difference  dimension 
which  measures  the  change  in  the 
individual  over  time. 

In  selecting  the  criterion  measure  to  be 
used  in  a  research  design  there  are  four 
basic  considerations  upon  which  to  evaluate 
the  criterion: 

(1)  Relevance. 

(2)  Reliability. 

(3)  Discrimination. 

(4)  Practicality. 

Criteria  measures  used  in  evaluation 
usually  fall  into  one  of  these  categories: 

(1)  A  test 

(2)  A  rating 

(3)  A  production  measure 

(4)  Some  archival  record  data 

For  training  evaluation  all  internal 
criteria  measures  are  some  form  of  test 
with  the  objective  of  indicating  the 
increase  in  knowledge  resulting  from 
training.  Rating  scales  are  used  to 
measure  total  Job  performance  but  are  not 
too  valid.  Productive  measures  increase 
validity  but  introduce  social  dynamics  of 
the  job  situation.  Archival  records  offer 
some  of  the  most  interesting  possibilities 
for  unusual  criteria  but  result  in  a  high 
degree  of  chaotic  error  and  systematic 
bias. 
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Requirements,  Specific 

3.1|  Performance 

Criteria,  Specific 
3.5  Me 'sure  lent  Procedures 


The  objective  of  this  study  was  to  support 
the  Manpower  Requirements  and  Resources 
Control  System  (MARRCS)  by  providing  an 
effective  means  of  assessing  benefits  and 
costs  in  management  and  information 
systems.  The  primary  focus  is  on  measuring 
benefit  of  the  information  rather  than 
cost.  The  cost  and  benefit  indices 
obtained  through  this  approach  are  being 
used  to  identify  development  opportunities 
and  priorities  and  to  permit  comparisons 
between  present  and  proposed  systems  for 
manpower  planning  in  terms  of  possible 
increase  in  effectiveness.  Scales  that  are 
based  on  this  information  benefit  model 
presented  below  make  up  part  of  a  larger 
instrument  that  is  the  primary  vehicle  for 
MARRCS  Phase  I  data  collection.  That 
instrument,  the  Manpower  Personnel  Planning 
Questionnaire,  is  included  in  the  report  as 
Appendix  A. 

The  benefit  model  developed  in  this  study 
is  as  follows: 


4.  Planning  Components 
of  the  Process 

4.1  Analytic  Methods 

4.2  Parameter  Determinations 

4.3  Apparatus  for  Testing 

4.4  Personnel  for  Testing 

4.5  Test  Plans 


Realized  Value  (Benefit)  s  P  x  R  x  U  where, 
P,  the  Potential  Contribution,  is  a  value 
attached  to  the  information  on  the  basis  of 
some  predetermined  set  of  specifications 
that  the  information  should  meet; 


5.  Application  Components 
of  the  Process 

5.1  Test  Execution 

5.2  Data  Analysis 

5.3  Findings  Interpretation 

5.4  Conclusions  and 
Recommendations 

6.  Further  Research  Areas 

6.1  Measurement  System 
Limitations 

6.2  Research  Potentials/ 
Priorities 

6.3  Research  Planning 


R,  the  Received  Value,  is  th^  portion  of 
potential  contribution  that  is  normally 
received  by  users  of  the  information; 

U,  the  Utilization  Value,  is  the  portion  of 
the  received  value  that  users  are  normally 
able  to  actually  apply  in  performing  their 
functions. 
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Values  for  P,  R,  and  U  are  developed 
through  the  questionnaire  administered  to 
both  producers  and  users  of  the 
information.  P  is  estimated  on  a  scale  of 
0  to  4  while  R  and  U  are  estimated 
percentages.  Although  producer  and  user 
estimates  usually  have  some  bias  in  them, 
the  two  perspectives  are  complementary  and 
the  technique  gains  much  of  its  strength 
from  having  both  perspectives. 


Preliminary  analyses  were  performed  using  a 
total  of  145  observations  that  were 
obtained  by  applying  the  data  gathering 
instrument  in  limited  parts  of  the  Navy's 
manpower  planning  system.  Because  of  data 
constraints  conventional  independence 
assumptions  of  statistical  techniques  such 
as  regression  and  analysis  of  variance  were 
not  satisfied.  However,  the  basic 
objective  of  this  preliminary  analysis  was 
to  obtain  an  indication  of  how  helpful  the 
hypothesized  benefit  factors  might  be. 
Indications  are  that  of  the  three  factors, 
Potential  Contribution  (P)  is  most  strongly 
related  to  overall  benefit.  The  Received 
Value  (R)  factor  showed  the  next  highest 
regression  coefficient  with  Utilization 
Value  (U)  being  least  related. 

It  was  concluded  that  the  approach 
presented  in  this  report  is  viewed  as  a 
significant  advancement  in  dealing  with  the 
problem  of  benefit  measurement  and  analysis 
in  humanistic  systems.  It  is  recommended 
that  a  programmatic  approach  to  this 
problem  area,  using  the  benefit  factors 
developed  in  this  preliminary  study  be 
employed. 
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1.  State  of  the  Art  Review 
of  the  Process 

1 . 1  General  System 
Measurements 

1.2  System  Taxonomy 
Model  (STM) 

1.3  Overall  Conceptual 
Process  Model  (CPM) 

2.  Contextual  Components 
of  the  Process 

2.1  System  Definition 

2.2  Mission  Definition 

2.3  Environment  Definition 
2.  A  General  Constraints 

2.5  Performance 
Requirements,  Ultimate 

2.6  Performance 
Criteria,  Ultimate 


1.1  The  field  of  systems  analysis  has 

traditionally  been  devoted  to  obtaining  the 
necessary  basis  for  design  or  redesign  of 
hardware  systems.  While  this  analytic 
capability  is  no  less  desirable  for  soft 
systems,  applications  in  that  area  have 
been  hampered  by  data  deficiencies, 
difficulties  in  system  definition  and  the 
specification  of  desired  performance, 
measurement  problems,  and  the  like.  As  a 
result,  many  attempts  to  perform  systems 
analyses  on  organizations,  for  example, 
have  resulted  in  lengthy  verbal  products 
too  inescapably  static  to  have  a  real-time 
impact  on  system  design. 


3.  Analytic  Components 
of  the  Process 

3.1  Practical  Measurable 
Attributes 

3-2  Practical  Attribute 
Measures 
3-3  Performance 

Requirements,  Specific 
3. A  Performance 

Criteria,  Specific 
3-5  Measurement  Procedures 

A.  Planning  Components 
of  the  Process 
A.1  Analytic  Methods 
A. 2  Parameter  Determinations 
A. 3  Apparatus  for  Testing 
A. A  Personnel  for  Testing 
A. 5  Test  Plans 

5.  Application  Components 
of  the  Process 

5.1  Test  Execution 

5.2  Data  Analysis  (2) 

5.3  Findings  Interpretation 
5. A  Conclusions  and 

Recommendations 


Assess  the  state  of  the  current  system 
at  the  onset  of  a  major  development 
effort  (i.e.,  establish  a 
baseline) . 

Reveal  deficiencies  in  the  Manpower 
System  and  probable  candidates  for 
further  R  4  D. 


1.3  In  response  to  particular  problems  in  the 
field  of  system  analysis,  a  Technique  for 
Interactive  Systems  Analysis  (TISA)  was 
developed.  TISA  is  a  computerized 
technique  for  conducting  systems  analysis 
in  a  conversational  mode  from  interactive 
terminals.  It  uses  networking  algorithms 
to  access  and  structure  system  descriptive 
data  from  computer  files.  Specifically, 
TISA  was  developed  in  order  to  perform  the 
systems  analysis  necessary  to: 

(1) 


6.  Further  Research  Areas 

6.1  Measurement  System 
Limitations 

6.2  Research  Potentials/ 
Priorities 

6.3  Research  Planning 
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(3)  Provide  a  convenient  and  effective 
means  for  assessing  the  progress  of 
the  development  program  at  any  point 
in  its  evaluation. 

3.3  TISA  was  designed  to  meet  certain 

requirements  of  data  manipulation.  These 

requirements  included  the  following: 

(1)  The  ability  to  structure  the  data  in  a 
network  flow  context. 

(2)  The  ability  to  structure  the  data  in 
an  organizational  context. 

(3)  Versatility  of  data  access  for  direct 
and  selective  analysis  with  a  variety 
of  routines  in  an  interactive  mode. 

(*0  Speed,  lucidity,  and  reliability  in 
the  graphical  depiction  of  any  system 
of  communications  as  specified  by  a 
selected  data  set. 

5.^  The  following  recommendations  are  made  for 

TISA: 

(1)  The  development  of  TISA  into  a 
standard  initial  analysis  approach  for 
soft  system  development  programs. 

(2)  The  use  of  TISA  as  a  methodological 
device  in  more  basic  areas  of  research 
such  as  organizational  behavior, 
information  processing,  and  human 
factors. 


Duning,  K.E.,  Hickok,  C.W.  Emerson,  K.C.  &  Clement,  W.F.  Control-display 
testing  requirements  study  (AFFDL-TR-72-122) .  Cedar  Rapids,  IA:  Collins 
Radio  Co.,  December  1972.  (AD-759  539). 


1.  sute  of  the  Art  Review  1.3  The  following  steps  represent  the  blocks  in 

or  the  Process  a  procedural  block  diagram  culminating  in 

1.1  General  system  system  and  pilot-centered  evaluation 

Measurements  Criteria: 

1.2  System  Taxonomy 
Model  (STM) 


1.3 

Overall  Conceptual 

Process  Model  (CPM) 

(1) 

Contextual  Components 

(2) 

of  the  Process 

2.1 

System  Definition 

(3) 

2.2 

Mission  Definition 

2.3 

Environment  Definition 

2.0 

General  Constraints 

2.5 

Performance 

Requirements,  Ultimate 

(4) 

2.6 

Performance 

Criteria,  Ultimate 

3.  Analytic  Components 


of  the  Process 

(5) 

3.1 

Practical  Measurable 
Attributes 

3.2 

Practical  Attribute 

Measures 

(6) 

3.3 

Performance 

Requirements,  Specific 

3-4 

Performance 

Criteria,  Specific 

(7) 

3.5 

Measurement  Procedures 

Planning  Components 
of  the  Process 

(8) 

<1.1 

Analytic  Methods 

(9) 

<1.2 

Parameter  Determinations 

4.3 

Apparatus  for  Testing 

0.0 

Personnel  for  Testing 

(10) 

4.5 

Test  Plans 

Application  Coeiponents 
of  the  Process 

5.1 

Test  Execution 

5.2 

Data  Analysis 

5.3 

Findings  Interpretation 

5.4 

Conclusions  and 
Recommendations 

Further  Research  Areas 

6.1 

Measurement  System 
Limitations 

6.2 

Research  Potentials/ 
Priorities 

6.3 

Research  Planning 

Describe  vehicle  operational  profile. 

Select  outcomes  of  interest. 

Specify  outcomes  and  pilot  acceptance 
in  terms  of  critical  limits  of 
pertinent  variables  in  numerical  terms 

Determine  system  error  and  state 
variable  performance  response  to 
inputs. 

Determine  outcome  probabilities  and 
pilot  acceptance  probabilities. 

Define  safety,  operational  capability 
and  pilot  acceptance  design  qualities. 

Determine  procedural  variables. 

Determine  task  variables. 

Determine  environmental  variables. 

Define  normal/degraded  feed  back 
arrangements  and  control-display 
mechanizations  to  perform  functions. 
Allocate  functions  to  manual  and/or 
automatic  systems. 
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(11)  Identify  system  performance-centered 
variables  and  physical 
characteristics. 

(12)  Identify  human  operator-centered 
variables. 

2.1  The  system  used  to  develop  control-display 
testing  requirements  is  a  Microwave  Landing 
System  (MLS).  The  MLS  provides  increased 
senior  capability  to  support  terminal  area 
approach  and  landing  operations.  Its 
potential  includes  zero  visibility 
landings,  maximum  runway  utilization,  noise 
abatement,  ground  fire  avoidance  and 
reduced  separation  standards. 

2.2  The  mission  to  be  evaluated  is  aircraft 
approach  and  landing  using  MLS. 

2.5  The  ultimate  performance  requirement  of  the 
MLS  is  to  insure  successful  aircraft 
landing. 

2.6  The  ultimate  criteria  of  the  MLS  system  is 
that  the  aircraft  must  be  within  a 
successful  landing  window  defined  by 
dispersions  at  decision  height  and 
reference  position  at  touchdown. 

3.1  Quantitative  and  qualitative  performance 

and  measures  and  evaluation  criteria  are  as 

3.2  follows: 

(1)  System  Performance  (Evaluation 

criteria  are  commensurate  with  metrics 
and  absolute  in  value) . 

(a)  At  Approach  Window 

.  Location  in  state  space  with 

respect  to  window  boundaries 

.  Probability  of  approach 

success 


A-104 


I Topics  Relevant 
I  to  System  Development  i Topic 
land  Evaluation  Technology  I  No. 


ABSTRACT 


(b)  At  Touchdown 


Longitudinal  and  lateral 
touchdown  location  >*ith 
respect  to  runway 

Sink  rate 


Sideslip 

Heading 

Pitch,  and  roll  attitudes 


,  Airspeed  error 

(c)  Composite  Measures 

(2)  Safety  Measures 

(a)  Probabilities  (Evaluation 
criteria  are  commensurate) 


.  Successful  landing 

,  Successful  missed  approach 

,  Accident  or  incident 

,  Margin  (stall,  performance, 

etc.) 

(b)  Qualitative  Assssments 

(Evaluation  criteria  are  relative 
and  subjective;  the  graceful 
degradation  hypothesis  provides  a 
guide) . 

,  Missed  approach  procedures 

,  Failure  detection  procedures 

.  Emergency  takeover  procedures 


(3)  Pilot  Performance  and  Acceptance 

Measures 

(a)  Pilot  Dynamic  Behavior 

(Evaluation  criteria  are  relative) 

.  Describing  functions  (loops 
closed  and  equalization 
demanded;  control  display 
associations  and  residual 
cross-coupling;  sensitivity 
of  stability,  disturbance 
regulation,  and 
command-following 
performance  to  variations  in 
gain,  time  delay,  and 
equalization;  the  adaptive 
feed  back  selection 
hypothesis  and  successive 
organization  of  perception 
hypothesis  provide  guides). 

.  Scanning  activity 

distributions  (incoherence 
in  system  performance  caused 
by  scanning  remnant;  system 
status  monitoring  threshold 
for  confidence  and 
decision-making;  the  display 
arrangement  hypothesis 
provides  a  guide). 

.  Opinion  ratings 

(psychometric  scales). 

.  Workload  and  operability 

assessment  (excess  control 
capacity;  auxiliary  task 
scores  and  loads; 
psychophysiological 
measurements;  there  is  no 
guide  to  evaluation  other 
than  sensitivity  and 
relative  differences. 

(b)  Pilot  Acceptance  of  System 

Performance 
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.  Attitude,  attitude  rate,  and 
load  factors  variances  from 
trinned  values  (Evaluation 
criteria  are  commensurate 
and  absolute,  e.g.a 
probabilities  of  exceeding 
acceptable  levels  from 
trimmed  values) . 

.  Control  displacement  and 

rate  variances  from  trimmed 
values  (Evaluation  criteria 
are  commensurate  and 
absolute,  e.g.,  probabili¬ 
ties  of  exceeding  maximum 
authorities) . 

.  Response  compatibility  and 

motion  harmony-automatic  and 
flight  director  versus 
manual  control  (Evaluation 
criteria  are  relative  to  the 
response  and  motion 
attributes  under  manual 
control) . 

,  Command  consistency-flight 

director  versus  Manual 
control  Evaluation  criteria 
are  based  on  the  consonance 
between  the  spectral 
distribution  of  status 
variables  in  the  director 
command  and  the  displayed 
status  variables 
themselves) . 

.  Qualitative  assessments- 

pilot  commentary  (Evaluation 
criteria  are  subjective  and 
relative) . 

4.1  The  control-display  design  validation 

process  will  be  conducted  in  three  phases: 


(1)  Theoretical  analysis 


!  Topics  Relevant 

1  1 

1  1 

to  System  Development 

iTopic! 

and  Evaluation  Technology 

I  No.  ! 

ABSTRACT 

(2)  Simulation 

(3)  Flight  test 

4.3  During  actual  flight  tests,  photo¬ 
theodolites  may  be  used  for  tracking 
simulated  IFR  approaches,  but  radar  must  be 
used  under  actual  IFR.  Special  flight 
control  systems  will  be  required  to  allow 
pilot  performance  measurement. 

4.5  Flight  test  plans  must  define  specific 
tasks  and  responses  as  well  as  complex 
sequences  of  tasks  designed  to  provide 
information  on  pilot  understanding, 
workload,  tracking,  monitoring,  and 
decision-making  abilities. 

In  the  area  of  tracking  performance, 
specific  acquisition,  approach,  and  missed 
approach  paths  need  to  be  defined,  and  data 
on  aircraft  displacement  and  attitude 
accumulated  through  the  beam  capture, 
approach,  and  landing  phases.  Data  from 
flights  will  be  used  to  form  statistical 
performance  distribution  curves  for 
comparative  analyses.  To  help  evaluate  one 
aspect  of  pilot  workload,  these  same 
flights  can  be  used  to  obtain  similar 
distribution  curves  for  wheel  motion, 
command  nulling  error,  pilot's 
eye-point-of-regard ,  and  subjective 
ratings.  When  combined  with  pilot 
interview  data,  these  data  will  form  an 
analytical  basis  for  control-display  system 
evaluation. 


In  addition  to  verifying  beam  capture, 
approach,  and  landing,  special  emphasis 
will  be  required  to  exercise  the  new 
capabilities  of  MLS  coverage  for  guiding 
and  monitoring  segmented  and  curved 
approach  paths  and  missed  approaches. 
Deliberate  disturbances  to  test  control 
responses,  distracting  influences  to  test 
pilot  error  sensitivity,  deliberate 
failures  to  test  pilot  reaction,  and 
extensive  use  of  the  real  environment  will 
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be  made  to  determine  display  capabilities. 

If  the  TIFS  aircraft  can  be  made  available, 
simulated  weather  would  be  a  possibility  to 
provide  one-to-one  correspondence  of 
vehicle  and  simulation.  Further,  multiple 
vehicles  can  be  simulated  with  the  TIFS 
vehicles.  This  would  provide  a  very 
flexible  platform,  requiring  only  one  set 
of  instrumentation. 

Use  of  the  digital  computer  and  flexible 
displays  may  be  strong  contributors  to  the 
hardware  flexibility  requirements.  This 
would  allow  study  of  data  rate  problems 
but,  more  significantly,  would  provide  a 
capability  of  readily  changing  control-dis¬ 
play  parameters  over  a  very  wide  range. 

Such  tools  should  certainly  be  considered 
for  curved  path  studies.  Controlled  inputs 
allowing  for  correlation  studies  in  the  air 
should  be  considered  with  the  flexibility 
provided  by  this  advanced  equipment  to 
measure  pilot  behavior. 

Automatic  data  logging  can  be  valuable  as  a 
flight  test  technique.  Failure  informa¬ 
tion,  data  discrepancies,  and  complex  path 
geometries  can  be  introduced  and  pilot 
responses  noted  with  experimental 
certainty.  Extensive  use  will  be  made  of 
the  monitor  pilot,  voice  recording,  strip 
chart  recording,  magnetic  tape  recording, 
and  controlled  test  stimuli. 

A  program  plan  for  control-display  testing 
requirements  for  the  new  Microwave  Landing 
System  can  expeditiously  and  effectively  be 
implemented  within  the  5-year  period.  A 
great  deal  of  directly  relevant,  applied 
research  on  landing  systems  has  been 
conducted  by  many  agencies.  Perhaps  the 
most  notable  example  is  the  Air  Force  PIFAX 
program.  For  well  over  a  decade  this  and 
corollary  efforts  have  addressed  key 
control-display,  instrumentation, 
operational,  and  pilot  factor  problems  in 


approach  and  landing  operations.  The 
result  is  a  large  reservoir  of 
experimentally  demonstrated  concepts, 
principles,  and  prototype  instrumentation. 
Also,  the  existence  of  various  ILS-based 
systems  for  almost  three  decades  has 
developed  a  broad  appreciation  for 
desirable  operational  characteristics  among 
users. 

It  remains  to  condense,  coalesce,  and  apply 
these  inputs  into  a  meaningful  and 
comprehensive  program  plan.  In  this 
section,  an  attempt  was  made  to  show  how 
theoretical  analysis,  simulation,  and 
flight  test  can  be  coordinated  so  that, 
when  all  levels  of  analysis  are  completed, 
the  resulting  control-display  system  design 
will  be  validated  as  "best." 
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1.  State  of  the  Art  Review  2.1 

of  the  Process 

1.1  General  Systen 
Measurements 

1.2  System  Taxonomy 
Model  (STM) 

1.3  Overall  Conceptual 
Process  Model  (CPM) 

2.  Contextual  Components 
of  the  Process 

2.1  System  Definition 

2.2  Mission  Definition 

2.3  Environment  Definition 

2.4  General  Constraints 

2.5  Performance 
Requirements,  Ultimate 

2.6  Performance  2.2 

Criteria,  Ultimate 

3.  Analytic  Components 
of  the  Process 

3.1  Practical  Measurable 

Attributes  2.3 

3.2  Practical  Attribute 
Measures 

3.3  Performance 
Requirements,  Specific 

3.4  Performance 
Criteria,  Specific 

3.5  Measurement  Procedures 

3.2 

4.  Planning  Components 
of  the  Process 

4.1  Analytic  Methods 

4.2  Parameter  Determinations 

4.3  Apparatus  for  Testing 

4.4  Personnel  for  Testing  -  - 

4.5  Test  Plans 

5.  Application  Components 
of  the  Process 

5. 1  Test  Execution 

5.2  Data  Analysis 

5.3  Findings  Interpretation  4.1 

5.4  Conclusions  and 
Recommendations 

6.  Further  Research  Areas 

6.1  Measurement  System 
Limitations 

6.2  Research  Potentials/ 

Priorities 

6.3  Research  Planning 


An  Integrated  Test  Facility  was  designed 
and  constructed  combining  the  individual 
test  courses  developed  in  a  previous  phase 
of  this  study.  Requirements  were  developed 
for  an  instrumentation  system  to  measure 
automatically  the  performance  of  test 
participants  and  requirements  and 
specifications  were  developed  for  a 
centralized,  computerized,  data  logging 
system  to  record,  process  and  statistically 
analyze  performance  data  collected  by  the 
system. 

A  field  methodology  was  developed  for 
measuring  the  effects  of  experimental 
clothing  and  equipment  on  the  combat 
effectiveness  of  individual  infantrymen. 

The  Integrated  Test  Facility,  located  at 
Camp  Picket,  Va,  consisted  of  eight  test 
situations,  five  of  which  were  developed 
and  tested  in  a  previous  phase  of  this 
study. 

Performance  data  were  collected  by  various 
types  of  electromechanical  sensor  devices 
located  on  the  various  performance  courses 
of  the  Integrated  Test  Facility. 

Sensors  were  activated  by  the  activities  of 
the  subjects  causing  input  signals  to  be 
sent  to  a  centrally  located  data  logging 
facility. 

The  Integrated  Test  Facility  consisted  of 
eight  distinct  courses  representative  of 
important  combat  tasks.  Estimates  of  the 
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performance  scores  which  each  course 
generated  were  collected  as  well  as 
estimates  of  the  time  required  to  complete 
each  course. 

5.4  The  tryout  and  evaluation  which  was 

necessary  to  meet  the  last  objective  of 
this  phase,  could  not  be  accomplished  due 
to  procurement  difficulties  in  acquiring 
the  Data  Logging  System.  The  evaluation 
and  final  refinement  will  be  conducted  upon 
acquiring  this  system. 
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The  system  evaluated  was  a  105  nro  howitzer 
battery  consisting  of  six  sections,  each 
with  one  howitzer  and  an  eight-man  team. 

The  tactical  activity  was  an  RSOP 
(Reconnaissance,  Selection  and  Occupation 
of  Position):  105  mm  howitzer  sections 
conducting  modified  planned  occupation  of 
an  area.  The  occupation  was  preceded  by  a 
recon  party  in  order  to  determine  firing 
positions,  installation  sites,  routes  of 
march,  defensive  plans,  etc.  Each  section 
was  to  complete  four  major  actions: 
uncoupling  the  howitzer,  preparing  for 
actions,  firing  and  inarch  order. 

Each  team  member  was  equipped  with  the 
following:  fatigue  uniform,  field  jacket, 
helmet  with  liner,  M56  harness  with  ammo 
pockets,  canteen,  entrenching  tool,  poncho, 
M16  rifle  and  M17  field  mask. 


2.4  The  field  test  had  the  following  conditions 
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The  characteristics  to  be  measured  were: 

(1)  Sequence  of  team  subtasks. 

(2)  Number  of  visual,  oral  and  manual  team 
coordination  cues. 

(3)  Quality  of  performance. 

(4)  Time  to  perform. 

(5)  Density  of  team's  activity. 

(6)  Number  of  critical  incidents. 

(7)  Subjective  rating  of  performance. 

(8)  Tactical  accuracy. 

(9)  Total  time  of  team  performance. 

(10)  Comparative  indications  of 
physiological  energy  expenditure. 

The  units  of  measurement  were  as  follows: 

(1)  Number  of  scheduled  and  unscheduled 
oral  communications. 

(2)  Number  of  scheduled  and  unscheduled 
visual  communications. 

(3)  Number  of  errors  by  team  and 
individual. 

(4)  Number  of  unsafe  conditions. 

(5)  Time  data  in  minutes  and  seconds. 

(6)  Quality  data  was  "good,"  "fair,"  or 
"poor,"  based  on  the  subjective 
judgment  of  an  experienced  battery 
officer. 

(7)  Heart  rate  of  certain  team  members. 
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3.5  Measurement  was  accomplished  using  a  series 
of  methods: 

(1)  Visual  recording  by  film  and  TV 
camera. 

(2)  Oral  recording  by  taping  and  pick-up 
technicians. 

(3)  Observers  for  visual  and  oral 
interactivity  and  team  events. 

4.1  The  data  analysis  methods  employed  were: 

(1)  Means 

(2)  Standard  deviations 

(3)  t  ratios 

(4)  Regression  analysis 

(5)  Sequence  effects  analysis 

(6)  After-task  questionnaires 

4.3  The  test  apparatus  included  film  and  TV 
camera,  tape  recorders,  still  cameras, 
telemetry  charts  (heart  rate). 

4.4  Forty-eight  test  subjects  were  used  (six 
teams  of  eight  men  each).  The  chief  of 
section  was  either  an  E-5  or  E-6.  All 
others  were  E-2  or  E-3's* 

5.2  The  resulting  analysis  pointed  out  vari¬ 
ables  relevant  to  small-group  interaction 
and  performance  which  were  able  to  dis¬ 
criminate  differences  between  performance 
of  teams  with  and  without  the  M-17 
protective  mask. 

5.4  The  conclusions  of  this  study  can  be 
summarized  as  follows: 
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(1)  A  simple  measurement  design  is 
inadequate  for  field  testing  of  teams. 
Observations  must  be  structured  to 
detect  separate  pitterns  of  perform¬ 
ance  across  multiple  variables, 
individual  team  members,  and  various 
sub-tasks. 

(2)  Human  observers  at  the  field  site 
cannot  alone  record  this  wide  range 
of  data  at  the  pace  of  team  perform¬ 
ance.  Data  recording  equipment, 
particularly  video-recorders,  assure  a 
more  adequate  and  objective  data 
collection. 

(3)  Performance  of  certain  team  tasks, 
generally  those  requiring  individual 
initiative  in  coordinating  activity, 
may  be  significantly  altered  by  the 
wearing  of  protective  masks.  Other 
tasks  of  a  repetitive  drill  nature  may 
be  much  less  affected.  Even  very 
meticulous  testing  during  field 
training  activity  may  fail  (because  of 
the  "slack  time"  and  indirect 
motivations  inherent  in  the  practice 
environment)  to  reflect  validly 
potential  decrements  in  a  tactical 
environment. 

(4)  For  105  mm  howitzer  crews 
specifically,  there  will  probably  be 
no  consistently  critical  performance 
problems  while  wearing  protective 
masks,  although  the  team's  normal 
pattern  of  oral  intercommunication 
is  definitely  disrupted. 

The  results  of  this  effort  offer  three 
promising  directions  for  further  study  in 
the  areas  of  field  collection  equipment, 
team  data  analysis,  and  standard 
performance  testing. 
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(1)  A  system  of  team  data  reduction  and 
analysis  in  depth  should  be 
sophisticated,  so  that  large 
quantities  of  data  from  mul' iple 
trials  over  time  could  be  methodically 
built  into  a  data  base  for  each 
variable,  team  member,  and  sub-task  of 
a  standard  test.  Application  of 
automatic  data  processing  and 
mathematical  modeling  should  be 
investigated,  and  weak  spots  in  the 
initial  design,  such  as  the  quality 
and  density  of  activity  variables, 
should  be  strengthened  through  quality 
control  and  systems  analysis 
techniques. 

(2)  The  generalizability  of  objective 
field-monitoring  techniques,  such  as 
video-recording,  should  be  explored  in 
detail.  Any  field  performance  that  is 
broadly  repeatable  could  become  a 
standard  test,  in  effect,  by  visually 
recording  each  trial  and  redirecting 
the  structuring  control  from  the 
participants  to  the  observation 
procedure:  standardly  structured 
camera  coverage,  videotape  editing  and 
playback,  observer  training,  and  video 
viewing  schedules  by  separate 
variables  and  sub-tasks. 

(3)  Routine  team  testing  should  be 
conducted  using  the  improved  data 
collection  devices  and  data  analysis 
techniques  perfected  in  the  study 
described  above.  The  current  study 
emphasized  the  detection  of  pattern 
changes,  not  the  significance  or 
tactical  validity  of  all  such  changes. 
In  its  field  observations,  this 
objective  was  achieved.  The  promising 
trends  in  variables  from  this  small 
number  of  trials  should  now  be  pursued 
in  a  much  larger  "n"  to  provide  the 
data  base  for  probable  statistical 
significance  in  several  categories. 
These  extended  performance  tests,  if 
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done  with  chemical  protective 
equipment,  could  then  possibly 
establish  reliable  patterns  of 
difference  valuable  in  improving 
design  of  the  equipment  and  in 
predicting  masked  team  behavior  under 
valid  tactical  conditions. 
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3.1  This  study  describes  methods  for  the 
formulation  of  measures  of  effectiveness 
(MOE's)  used  in  studies  where  variables  are 
not  readily  (if  at  all)  measurable  in  the 
real  world.  Within  this  report  is  a  survey 
(and  listing)  of  both  algebraic  and 
statistical  techniques  which  are  useful  in 
bounding  and  estimating  MOE's.  Bounding  in 
this  context  means  confidence  limits  or 
intervals. 

3.2  The  results  of  the  survey  are  intended  to 
be  illustrative,  rather  than  exhaustive  of 
some  of  the  types  of  MOE's  in  use.  For 
example,  an  MOE  for  Airborne  Antisubmarine 
Warfare  (ASW)  is  the  probability  of 
submarine  detection  by  helicopter.  These 
MOE's  are  chosen  from  a  spectrum  of  naval 
warfare  areas  with  particular  emphasis 
placed  on  antisubmarine  warfare. 

3.5  For  each  MOr  the  measurement  techniques  are 
provided  to  determine  the  bounds  of  the  MOE 
values.  These  techniques  are  of  various 
forms  such  as  linear-exponential  product, 
sum  of  exponentials,  sum  of  products  etc. 
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Frequency  —  the  absolute  and  relative 
frequency  for  each  of  the  tasks  for 
given  periods  of  time  under  any  type 
of  loading  condition  for  any  type  of 
operator. 

Time  —  mean  and  relative  times  for 
each  task,  under  conditions  of  varying 
loads  and  operators. 


1.1  This  document  describes  a  procedure  for 
collecting  performance  data  and  examining 
man/machine  interaction.  This  procedure, 
called  a  Task  Analysis  Reduction  Technique 
(TART),  allows  for  facilitation  of  human 
performance  quantification,  clarification 
of  analysis  and  improved  useability  of  the 
data. 

3.1  The  TART  focuses  on  the  interface  at  the 
task  level  of  complexity.  Analysis  at  the 
task  level  is  advantageous  because  one  can 
according  to  one's  needs,  break  the  task 
down  into  movements  and  actions  for 
detailed  analysis  or  one  can  combine  tasks 
and  analyze  different  configurations  at  a 
higher  functional  level. 

3.2  The  example  provided  demonstrates  the  use 
of  TART  by  deriving  measures  for  the  air 
detector/tracker  (ADT)  position  on  the 
Antisubmarine  Warfare  Tactical  Data  System 
(ASWTDS) . 

3.3  The  TART,  in  general,  can  produce  the 
following  types  of  data: 

(1) 
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(3)  Frequency/time  comparisons  — 
comparisons  between  the  absolute  and 
relative  occurrence  of  each. 

(4)  Sequential  properties  —  the  nature  of 
strategies  employed  by  the  various 
operators  in  performing  their  various 
functions . 

(5)  Error  analysis  —  number  and  type  of 
deviations  from  expected  procedures 
and  the  effects  of  innovative  actions. 

Specific  performance  measures  are  presented 
in  the  ASWTDS  example. 

The  TART  consists  of  a  three-part  procedure: 

(1)  Task  analysis  Operational  Sequence 
Diagrams  (OSDs)  are  developed  for  the 
tasks  observed. 

(2)  A  video  recording  of  the  action  and 
feedback  channels  of  the  machine  is 
obtained . 

(3)  The  video-recording  of  that  activity 
is  reduced  to  a  time-line  sequence  of 
basic  tasks,  and  an  analysis  of  the 
frequency,  time  and  sequential 
properties  of  that  time-line  is 
performed . 

Using  the  techniques  described  in  this 
report,  it  is  possible  to  collect  and 
quantify  useful  human  performance  data.  It 
can  be  used  to  analyze  a  realistic 
man/machine  interface.  An  automated  form 
of  the  task  analysis  OSD  is  useful  for 
developing  base-line  performance 
expectations,  and  the  basic  form  of  the 
data  collected  can  be  useful  types  of 
analyses. 

It  is  recommended  the  TART  system  be 
continued  and  onboard  ship  programs  begun. 

It  was  also  suggested  that  this  technique 
be  included  in  the  repertoire  of  those 
considering  man/machine  interface  analysis, 
and  a  data  bank  of  empirical  personnel 
performance  data  begun. 
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The  purpose  of  this  paper  is  to  consider 
data  concerning  the  relationship  of  a  task 
taxonomy  and  performance  measurement 
taxonomy.  A  task  classification  scheme  was 
used  as  follows: 

(1)  Learning  identifications. 

(2)  Perceptual  discriminations. 

(3)  Principles  and  relationships 
comprehension. 

(H)  Procedural  sequencing. 

(5)  Decision  making. 

(6)  Perceptual  motor  skills. 

The  performance  measurement  classification 
system  is  categorized  on  the  basis  of 
remoteness  from  actual  job  performance. 

The  four  major  segments  along  this 
continuum  are: 

(I)  On-the-job  measures. 

(2)  Work  sample  measures. 

(3)  Simulated-job  measures. 

(H)  Correlated-job  measures. 
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2.1  The  system  studied  was  that  of  the  general 
vehicle  mechanic  for  wheeled  and  tracked 
army  vehicles. 

4.1  A  four-day  proficiency  test  consisting  of 
33  sample  exercises  was  used.  The  test 
included  a  diagnostic  scoring  procedure  for 
use  in  scoring  men  on  quality  of 
performance.  In  addition  a  questionnaire 
was  used  to  obtain  supplemental  information 
regarding  the  mechanic's  experience.  A 
second  phase  of  research  dealt  with  a 
comparision  of  two  job  correlated  measures 
with  the  work  sample  via  a  paper  and  pencil 
evaluation  test. 

4.4  The  test  subjects  were  38  organizational 
mechanics  drawn  from  all  organizational 
maintenance  units  at  Fort  Knox.  The 
exercises  we»*e  individually  scored  by 
experienced  mechanics  trained  in  test 
administration  procedures. 

5.2  The  results  indicated  that  the  total  test 
appears  to  have  a  high  degree  of 
reliability,  indicating  it  should  permit  a 
high  degree  of  accuracy  measurement  when 
used  as  a  criterion  in  evaluating  other 
measurement  techniques.  The  second  phase 
written  test  had  a  low  degree  of  validity 
when  correlated  with  the  work  sample  and  as 
a  result  should  not  be  used  in  group  or 
individual  measurement. 

5.4  Further  research  should  be  conducted  along 
two  parallel  lines: 

(1)  Development  and  refinement  of  an 
interim  task  classification  system. 

(2)  Development  and  refinement  of  a 
interim  classification  system  for 
human  performance  measures. 
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1.1  This  dissertation  treats  the  problem  of 
designing  high-performance,  closed-loop 
control  systems  which  include  a  human 
operator.  In  particular  it  describes 
research  conducted  to  answer  the  following 
rather  general  questions: 
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3.1  Practical  Measurable 
Attributes 
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Measures 

3.3  Performance 
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3 .  H  Performance 

Criteria,  Specific 
3.5  Measurement  Procedures 


(1)  How  should  the  state  of  the  system  be 
displayed  to  the  operator  and  how 
should  his  responses  to  the  display  be 
processed  before  they  are  input  to  the 
system? 

(2)  In  what  way  does  the  operator's 
dynamic  response  limit  the  performance 
of  the  overall  system,  and  how  does 
this  limitation  depend  on  which  of  his 
several  possible  outputs  is  used  for 
controlling  the  system? 

The  results  presented  are  applicable  to  the 
design  of  a  wide  variety  of  manned  control 
systems. 
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5.  Application  Components 
of  the  Process 

5.1  Test  Execution 

5.2  Data  Analysis 

5-3  Findings  Interpretation 
5.4  Conclusions  and 
Recommendations 


6.  Further  Research  Areas 

6.1  Measurement  System 
Limitations 

6.2  Research  Potentials/ 
Priorities 

6.3  Research  Planning 


Erickson,  R.A.  Field  evaluation  of  a  visual  detection  model.  In  G.  W.  Levy 
( Ed . ) ,  Symposium  on  applied  model  of  man-machine  systems  performance 
(NR69H-$91 ) .  Columbus  OH:  North  American  Aviation,  November  1968.  (AD-697 

939). 


1.  State  of  the  Art  Review 

of  the  Process 

1 . 1  General  System 
Measurements 

1.2  System  Taxonomy 
Model  (STM) 
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4.5  The  report  describes  a  1962  field 
experiment  conducted  to  validate  a 
mathematical  model  of  the  visual  detection 
process.  All  observations  were  made  by 
pilots  flying  A-4  aircraft  above  a 
bulldozed  strip  in  the  desert.  Ground 
targets  were  a  Sherman  tank  and  a  radar  van 
without  the  radar  dish/antenna.  Thus,  the 
visual  search  was  in  one  dimension  only; 
the  model  was  not  capable  of  handling 
2-dimensional  search.  Flights  were 
conducted  at  altitudes  of  1000,  2500,  and 
4000  feet,  at  indicated  airspeeds  of  275, 
270,  and  265  knots,  respectively. 

5.3  Data  showed  that  the  model's  predictions  of 
detection  range  are  quite  accurate,  but  the 
prediction  of  recognition  range  is  in  error 
by  substantial  amounts. 
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1.  State  of  the  Art  Review  1.2 

of  the  Process 

1.1  General  System 
Measurements 

1.2  System  Taxonomy 
Model  (STM) 

1.3  Overall  Conceptual 

Process  Model  (CPM)  2  5 

2.  Contextual  .^ronents 
of  the  Process 

2.1  System  Definition 

2.2  Mission  Definition 

2.3  Environment  Definition 

2.4  General  Constraints 

2.5  Performance 
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2.6  Performance 
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3.  Analytic  Components 
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3. 1  Practical  Measurable 
Attributes 

3.2  Practical  Attribute 
Measures 

3.3  Performance 
Requirements,  Specific 

3.4  Performance 
Criteria,  Specific 

3.5  Measurement  Procedures 

3.2 

4.  Planning  Components 
of  the  Process 

4.1  Analytic  Methods 

4.2  Parameter  Determinations 

4.3  Apparatus  for  Testing 

4.4  Personnel  for  Testing 

4.5  Test  Plans 


The  present  research  described  a  series  of 
studies  conducted  to  develop  an  instrument 
in  terms  of  which  the  stimulus,  procedural 
and  response  characteristics  of  tasks  could 
be  described. 

The  basic  steps  in  this  research  were  to: 

(a)  develop  descriptive  characteristics  of 
tasks;  (b)  assess  the  reliability  of  rating 
scales  derived  to  measure  these  characteris¬ 
tics;  and  (c)  determine  if  these  character¬ 
istics  represented  correlations  of 
per formance. 

Major  components  of  a  task  were  identified 
and  treated  as  categories  within  which  to 
devise  task  characteristics  or  descrip¬ 
tions.  Each  characteristic  was  cast  into  a 
rating  scale  format  which  presented  a  defi¬ 
nition  of  the  characteristic  and  provided  a 
seven-point  scale  with  defined  anchor  and 
mid-points,  along  with  examples  for  each 
point. 

Nineteen  scales  were  developed  and 
evaluated  in  a  series  of  three  reliability 
studies.  The  following  are  the  task 
components  and  related  characteristics  (for 
which  the  19  scales  were  formed): 
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Component 

Goal 


Response 


Procedures 


Stimulus 


Stimulus- 

responses 


Task  Characteristics 

Number  of  output  units 

Duration  for  which  an  output 
unit  is  maintained 

Number  of  elements  per  output 
unit 

Work  load  imposed  by  task  goal 
Difficulty  of  goal  attainment 
Precision 
Rate 

Simultaneity  of  responses 

Amount  of  muscular  effort 
involved 

Number  of  steps 

Dependency  among  procedural 
steps 

Adherence  to  procedures 
Procedural  complexity 
Variability 
Duration 

Regularity  of  stimulus 
occurrence 

Degree  of  operator  control 

Reaction  time/ feedback  lag 
relationship 

Decision  making 
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4.1  The  paradigm  used  to  determine -whether  the 
task  characteristics  were  correlates  of 
performance  upon  which  predictive  relation¬ 
ships  might  be  established  was  that  of 
"post-diction" . 

4.2  Post-diction  referred  to  the  situation  in 
which  performance  measures  were  abstracted 
from  studies  already  existing  in  the 
literature. 

4.5  Subjects  were  supposed  to  rate  descriptions 
of  the  tasks  used  in  these  studies  on  task 
characteristics  scales  and  then  these 
ratings  were  to  be  subjected  to  multiple 
regression  analysis  to  establish  the  extent 
to  which  they  were  related  to  the 
performance  in  question. 

5.1  Two  such  post-diction  studies  were 
conducted. 


5.2 


5.3 


In  general,  it  was  found  that  a  subset  of 
scales  having  adequate  reliability 
consistently  emerged  in  all  three 
reliability  studies.  The  results  of  the 
two  post-diction  studies  were  encouraging 
in  that  significant  multiple  correlations 
of  .82  and  .73  were  obtained  between  task 
characteristic  ratings  and  the  performance 
measures . 


It  does  appear  possible  to  describe  tasks 
in  terms  of  task-characteristic  language 
which  is  relatively  free  of  the  subjective 
and  indirect  descriptions  found  in  other 
systems.  Further,  task  characteristics  may 
represent  correlates  of  performance. 
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Research  Planning 

The  results  of  a  feasibility  study  for 
determining  a  small  arms  measure  of 
effectiveness  for  handling  characteristics 
were  presented  in  this  report. 

The  system  being  measured  was  the  small 
arms  user,  and  .45  and  .38  caliber  pistols. 

The  handling  characteristics  of  these 
weapons  were  selected  for  evaluation. 

The  tests  were  conducted  on  a  typical 
pistol  range  common  to  most  Army 
installations. 

The  ultimate  performance  requirement  is 
that  the  small  arms  user  survives  in  a 
combat  or  other  situation  and  that  his 
weapons  are  utilized  effectively. 

The  ability  to  shift  a  weapon  rapidly  and 
accurately  from  one  target  to  another  is  a 
critical  factor  in  the  use  of  small  arms 
weapons. 

The  primary  purpose  of  this  study  was  to 
measure  an  individual's  performance  while 
he  was  accomplishing  a  series  of  movement 
and  firing  tasks  and  determine  whether  the 
resulting  MOE's  could  discriminate  between 
the  two  systems. 
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3.2  The  time  it  took  for  an  individual  to 
accomplish  these  tasks  without  error  was 
recorded.  In  addition,  an  index  of 
difficulty  for  the  performance  of  shooting 
tasks  was  developed  using  information 
theory. 

4.1  Analytic  methods  were  used  to  determine  if 
there  is  a  statistically  significant 
difference  between  the  mean  times  it  takes 
an  individual  to  complete  a  set  of 
prearranged  movement  tasks  with  two 
differently  handling  weapon  systems. 

4.3  The  weapons  used  in  this  experiment  were 
the  M1911A1  .45  caliber  automatic  pistol 
and  the  Smith  and  Wesson  Model  10-38 
caliber  revolver  and  appropriate 
ammunition.  Targets,  timing  devices,  and 
record  sheets  were  utilized  in  this 
experiment . 

4.4  The  subjects  were  selected  from  the 
population  of  military  personnel  who,  at  a 
minimum,  would  be  required  as  part  of  their 
combat  equipment,  to  carry  and  be  prepared 
to  use  a  sidearm.  All  participants  were 
volunteers,  ages  28-37,  all  were  officers, 
time  in  military  service  ranged  from  7-16 
years.  Range  of  experience  with  weapons  in 
question  varied. 

4.5  The  weapon,  task  sequence,  and  subject 
factor  levels  were  set  prior  to  conduct  of 
experiment.  Four  task  sequences  were 
selected.  Subjects  were  briefed  prior  to 
the  experiment  and  received  written 
instructions.  Practice  on  both  types  of 
weapons  was  permitted  followed  by  actual 
firing  in  the  task  sequence  assigned.  At 
the  end  of  the  firing  the  subject  filled 
out  an  information  sheet  giving  personal 
data  and  weapon  evaluation. 

5.3  The  conclusions  were  as  follows:  (a)  There 
was  no  statistical  difference  between 
weapons  used,  (b)  Three  of  the  four  task 
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sequences  were  different.  Sequences  1  and 
2  tested  as  having  no  significant 
difference,  (c)  The  average  information 
processing  rates  for  each  weapon  were 
different  but  were  not  tested  for 
statistical  significance.  The  index  of 
difficulty  measurement  of  the  tasks  was 
shown  to  have  a  high  correlation  tc  the 
time  required  by  each  weapon  to  perform  the 
tasks  that  had  one  or  more  movements 
specifically  designed  into  the  tasks. 
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(2) 

4.  Planning  Components 
of  the  Process 

4.1  Analytic  Methods 

4.2  Parameter  Determinations 

4.3  Apparatus  for  Testing 

4.4  Personnel  for  Testing 

4.5  Test  Plans 

5.  Application  Components 
of  the  Process 

5.1  Test  Execution 

5.2  Data  Analysis 

5.3  Findings  Interpretation 

5.4  Conclusions  and 
Recommendations 


A  subjective  mission  success  score  - 
the  instructor  pilot  rated  the  sub¬ 
ject’s  navigational  effectiveness  in 
terms  of  three  categories:  complete 
failure,  partial  success,  and  complete 
success. 

An  objective  mission  success  score  - 
this  measure  was  constructed  to  serve 
as  a  component  metric  representative 
of  the  subject’s  scores  on  four 
individual  measures:  number  of 
initial  points  (beginning  of  a  route) 
missed,  number  of  landing  zones 
missed,  number  of  250  meter  excursions 
from  the  course  line,  and  number  of 
1,000  meter  excursions. 


1.1  The  basic  objective  for  this  series  of 
studies  was  to  obtain  empirical  data  on  how 
the  Nap-of-the-Earth  (NOE)  navigational 
skill  level  of  Army  aviators  is  affected  by 
pilot  experience  and  two  levels  of 
training.  Other  objectives  were  to  define 

a  baseline  on  pilot  navigation  proficiency 
and  to  develop  a  field  research  methodology 
to  measure  pilot  performance  in  NOE  flight. 

2.6  The  dependent  measures  were  measures  of 
navigational  performance. 

3.1  Two  measures  of  mission  success  were  used, 
and  These  were: 

3.2 

(1) 


6.  Further  Research  Areas 

6.1  Measurement  System 
Limitations 

6.2  Research  Potentials/ 
Priorities 

6.3  Research  Planning 
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4.3  Two  UH-1H  helicopters  were  employed  in  the 
study  over  a  test  range  near  Troy,  Alabama. 

4.4  The  subjects  in  the  study  were  35  Army 
helicopter  pilots  who  were  currently 
proficient  with  the  test  copters  and  had 
some  exposure  to  NOE  flight  at  either  entry 
or  unit  level.  Four  Army  navigators 
conducted  the  tests,  two  being  highly 
qualified  NOE  instructor  pilots. 

4.5  The  navigators  were  assigned  missions  in 
which  designated  landing  zones  had  to  be 
found  for  simulated  medical  evacuations  or 
supply  deliveries.  All  35  aviators 
navigated  at  least  six  NOE  routes  ranging 
from  23  to  25  kilometers  (km)  in  length. 
Twenty-eight  of  the  aviators  were  also 
tested  on  aircraft  control  and  the 
performance  of  various  NOE  maneuvers. 

5.3  NOE  navigation  is  a  trainable  skill. 
Experienced  aviators  with  training 
performed  better  than  experienced  aviators 
without  training.  In  addition,  recent 
graduates  with  15  hours  of  NOE  training 
performed  better  than  experienced  aviators 
without  training. 

Specialized  training  in  NOE  navigation  is 
valuable.  The  instructor  pilots  indicated 
that  experience  is  more  important  in 
aircraft  control  than  in  navigation.  The 
group,  with  an  average  of  1,380  flight 
hours,  controlled  their  aircraft  better 
than  the  group  with  200  hours  but  this 
difference  in  experience  appeared  to  have 
no  effect  on  the  navigation. 

The  field  research  measures  and  techniques 
developed  for  these  experiments  seem  well- 
suited  for  NOE  navigation  research.  The 
Objective  Mission  Success  Score  (OMSS),  a 
composite  performance  measure  indicating 
the  probability  that  the  mission  would  be 
successfully  completed,  proved  useful  and 
has  a  .75  correlation  with  subjective 
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ratings  by  expert  NOE  instructor  pilots. 

As  expected,  it  was  shown  that  NOE 
navigation  is  an  extremely  difficult  and 
complex  task.  The  OMSS'  show  that  NOE 
missions  flown  over  the  Fort  Rucker  terrain 
resulted  in  an  overall  probability  of 
success  of  .63. 


A-134 


Finley,  D.L.  4  Muckier,  F.A.  Human  factors  research  and  the  development  of  a 
manned  systems  applications  science:  The  system  sampling  problem  and  a 
solution.  Northridge,  CA:  Manned  Systems  Sciences,  Inc.,  July  1976.  (AD-029 

V7) 


i  Topics  Relevant 

1  1 

1  1 

i  to  System  Development 

iTopic! 

land  Evaluation  Technology 

i  No.  i 

ABSTRACT 

1.  State  of  the  Art  Review 
of  the  Process 

1.1  General  System 
Measurements 

1.2  System  Taxonomy 
Model  (STM) 

1.3  Overall  Conceptual 
Process  Model  (CPM) 

2.  Contextual  Components 
of  the  Process 

2.1  System  Definition 

2.2  Mission  Definition 

2.3  Environment  Definition 

2 .4  General  Constraints 

2.5  Performance 
Requirements,  Ultimate 

2.6  Performance 
Criteria,  Ultimate 

3.  Analytic  Components 
of  the  Process 

3.1  Practical  Measurable 
Attributes 

3.2  Practical  Attribute 
Measures 

3.3  Performance 
Requirements,  Specific 

3. A  Performance 

Criteria,  Specific 
3.5  Measurement  Procedures 


1.2  This  report  points  out:  the  need  to 
identify  and  incorporate  systems  design  and 
operation  parameters  into  research 
programs;  the  nature  of  systems  research 
and  the  dimensional  problem,  and  presents  a 
model  to  support  the  systems  dimensionaliza- 
tion  process,  i.e.,  a  systems  taxonomy 
model . 

2.1  This  study  addresses  the  manned  system  and 
the  need  to  develop  a  body  of  knowledge  on 
the  dimension  of  a  system,  as  opposed  to 
such  components  as  an  individual  operator 
or  a  piece  of  equipment. 

5.3  The  beginnings  of  a  systems  taxonomy  model 
are  presented.  It  consists  of  three  major 
levels:  (a)  system  objectives;  (b)  system 
functional  purpose  and  (c)  system  charac¬ 
teristics  (structural  operator/equipment, 
operating  and  support  requirements).  These 
three  levels  are  further  defined  by  their 
relationship  to  the  nominal  versus  relative 
levels  of  measurement. 


4.  Planning  Components 
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A.i  Analytic  Methods 
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*1.3  Apparatus  for  Testing 
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Information  is  presented  on  how  to  use  this 
model  and  of  the  model  level  implications 
and  importance. 
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5. A  Conclusions  and 
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LEVELS  LEVELS  TAXONOMIC  CATEGORIES  A  DIMENSIONS 


LEVEL 

ONE 

Nominal  Measurement 

SYSTEM  OBJECTIVES 

•  Production 

•  Supply 

•  Navigation 

•  Air  Traffic  Control 

•  Health  &  Welfare 

•  Transportation 

•  Maintenance 

•  Weapons 

•  Surveillance 

•  Etc. 

LEVEL 

Nominal 

A 

SYSTEM  FUNCTIONAL  FURPOSES 

Nominal 

•  Indirect  command/control/guidance 
operations 

•  Relatively  direct  control 'navigation 
operations 

•  Maintenance  operations 

•  Data  or  materials  processing 

TWO 

u 

Relative 

V 

Relative 

•  Command 

•  Control 

•  Information 
.  Data 

LEVEL 

Relative  Measurement 

STRUCTURAL  CH ARACTERIST1CS 

.  Organization  and  layout 

•  Size 

•  Level  of  automation 

.  Implementation  capabilities 

OPERATOR /EQUIPMENT 
CHARACTERISTICS 

•  Human  skills,  equipment  conditions 

•  Human  abilities  &  IQs,  equipment 
capabilities 

•  Values 

•  Needs 

THREE 

(Ordinal.  Interval  and 
Ratio) 

OPERATING  CHARACTERISTICS 

•  Inputs  to  operator 

•  Operator  processing 

•  Operator  outputs 

•  Units  being  dealt  with  by  system 

•  Environment 

•  Feedback 

SUPPORT  REQUIREMENTS 
CHARACTERISTICS 

•  Materials  (including  people) 

•  Maintenance  (including  people) 

From  Finiey  et  al.  (1975) 


Systems  Taxonomy  Model 
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Over  the  pas  three  decades  there  has  been 
an  increasing  demand  for  quantitative 
techniques  of  human  performance  prediction 
in  man-machine  system  tasks.  A  somewhat 
bewildering  variety  of  methods  has  evolved 
to  satisfy  this  need,  ranging  from  specific 
task  simulation  to  classical  tests  of 
fundamental  human  abilities. 

From  the  existing  literature,  75  behavioral 
dimensions  were  defined  and  incorporated 
into  a  Performance-Descriptor  X 
Physical-and-Interactional-Categories 
Matrix.  This  is  shown  in  Figure  1. 

Implicit  in  the  adoption  of  the  dimensional 
approach  to  human  performance  prediction 
was  the  assumption  that  it  would  be 
possible  to  denote  a  set  of  specific 
procedures  which  would  define  a 
comparatively  objective  mapping  process 
wherein  the  accuracy  of  the  mapper  would  be 
more  a  function  of  available  knowledge  than 
of  the  goodness  of  his  intuition. 

(Mapping,  as  used  here,  refers  to  the  a 
priori  selection  of  those  socio- 
psychological  dimensions  which  would  be 
required  to  perform  an  operational  task.) 

The  basic  objective  of  this  program  was  to 
critically  review  tests  and  test  techniques 
for  human  performance  prediction.  Such  a 
review,  however,  is  best  facilitated  by 
conceptual  and  methodological  criteria.  At 
a  very  basic  level,  four  fundamental 
questions  were  asked: 

(1)  To  predict  what? 
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(2)  Upon  what  dimensions  and  "measures? 

(3)  With  what  tools? 

(4)  For  what  purposes? 

Human  performance  prediction  in  man-machine 
systems  must  be  concerned  with  three  levels 
of  measurement  analysis: 

(1)  System  requirements  and  appropriate 
system  performance  measurement. 

(2)  Human  operator  task  analysis  and  the 
performance  measures  related  to  that 
level. 

(3)  Basic  behavioral  dimensions  involved 
in  human  task  performance. 

Further,  the  precise  interrelationships 
between  these  levels  should  be  quantified. 

Tests  must  be  related  to  human  performance 
dimensions  found  in  human  operator  tasks 
which  are  executed  to  help  achieve  system 
performance  criteria.  For  tests  to  be 
meaningful  in  man-machine  systems, 
quantitative  transformations  must  be 
possible  between  levels.  This  required 
mapping  operation  turns  out  to  be  a 
formidable  technical  challenge. 

Both  the  questions  and  levels  of  analysis 
can  be  combined  into  a  single  conceptual 
structure,  as  shown  in  Figure  2.  The 
question  of  purpose  is  external  to  this 
matrix,  but  each  of  the  first  three 
questions  can  be  asked  at  each  of  the  three 
levels.  The  addition  of  an  analytic 
requirement  to  interrelate  these  levels 
results  in  a  Generalized  Methodological 
Model  which  can  be  (1)  used  to  evaluate  the 
existing  literature  and  (2)  form  a  frame¬ 
work  of  requirements  for  future  test  devel¬ 
opment.  To  be  general,  one  must  at  least 
postulate  a  hierarchical  system  structure 
consisting  of  many  levels  of  embedded 
functional  units,  e.g.,  total  system, 
subsystems/modules/etc. . .components,  action 
elements.  It  may  be  uncommon  to  find  a 
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complex  system  with  such  a  clear-cut 
hierarchical  structure;  many  system 
functional  units  may  depend  upon,  or 
influence,  many  o*  .ers.  Consequently,  the 
system  organization  may  require  a  complex 
block  diagram  to  display,  but  such  a  block 
diagram  will  also  be  required  for  other 
system  engineering  purposes. 

2.2  The  analysis  was  used  in  the  detailed 
application  of  the  human  performance 
prediction  methodology  (developed  from  the 
approach  implied  by  Figure  1)  to  a  specific 
man-machine  system  activity:  the  celestial 
and  space-object  radiometry  experiments 
conducted  during  the  Gemini  V  and  the 
Gemini  VII  missions. 

4.1  Several  intensive  analyses  were  performed 
at  several  levels  to  provide  specific 
answers  to  the  above  basic  questions.  The 
analytic  outputs  were  in  terms  capable  of 
quantitative  measure  and  the  relationships 
between  system,  system-man  and  the  human 
operator  levels  of  critical  performance 
were  identified  with  respect  to  these 
terras. 

6.1  Three  major  problems  were  found  to  dominate 
the  literature  on  human  performance 
prediction  tests  in  man-machine  system 
performance;  first,  elementary  and 
essential  rules  in  test  development  have 
been  frequently  ignored;  second,  modern 
techniques  for  the  development  of 
cost-effective  tests  and  test  batteries 
through  utility  analysis  have  not  been 
used;  and  third,  the  basic  issues  in  test 
validity  have  been  avoided.  So  long  as 
this  "strategy"  continues,  the  literature 
will  be  extremely  suspect.  However,  all  of 
these  difficulties  can  be  resolved. 

Many  methods  of  task  analysis  exist  within 
the  literature,  but  a  very  thorough  review 
of  these  methods  failed  to  reveal  any 
particular  method  of  direct  usefulness, 
showed  the  lack  of  standardization  in  the 
field,  and  suggested  t.iat  a  new  attempt  at 
a  basic  taxonomy  was  in  order. 
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As  a  result  of  trying  out  three  different 
behavioral  taxonomies,  it  was  found  that 
the  effectiveness  and  usefulness  of  a 
taxonomy  is  a  function  of  the  following 
factors: 

(1)  How  appropriate  the  level  of  detail  is 
to  the  purpose  of  the  taxonomy. 


(2)  How  cleanly  separated  and  appropriate 
the  categories  are. 

(3)  How  objectively  and  thoroughly  the 
categories  are  defined. 

(4)  In  the  case  of  the  analytic  behavioral 
categories,  how  completely  the 
taxonomy  covers  the  behavioral  domain. 


h  ! 
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1.1  The  paper  describes  an  application  of  a 

digital  computer  simulation  model  aimed  at 
predicting  maintenance  task  performance  in 
a  sonar  system  under  development.  The 
model  provides  for  simulation  of  one  or  two 
operators  in  a  man-equipment  interface 
situation. 


2.  Contextual  Components 
of  the  Process 

2.1  System  Definition 

2.2  Mission  Definition 

2.3  Environment  Definition 

2.4  General  Constraints 

2.5  Performance 
Requirements,  Ultimate 

2.6  Performance 
Criteria,  Ultimate 


2.1  The  application  described  involved  a 

simulation  of  the  coordinated  activities  of 
a  sonar  technician  and  sonar  supervisor  in 
performing  two  representative  maintenance 
tasks.  The  simulation  examined  the  impact 
of  different  skill  levels  and  time 
constraints  on  task  performance. 


3.  Analytic  Components  j,  ~ 

of  the  Process  * 

3.1  Practical  Measurable 
Attributes 

3.2  Practical  Attribute 
Measures 

3-3  Performance 

Requirements,  Specific 
3.1)  Performance 

Criteria,  Specific 
3.5  Measurement  Procedures 

5.3 

4.  Planning  Components 
of  the  Process 

4.1  Analytic  Methods 

4.2  Parameter  Determinations 

4.3  Apparatus  for  Testing 

4.4  Personnel  for  Testing 

4.5  Test  Plans 


To  employ  the  model,  the  task  to  be 
simulated  must  be  broken  down  into  subtasks 
(e.g.,  "depress  pushbutton,"  "set  toggle 
switch,"  etc.).  Input  parameters  define 
time  constraints,  probability  of  successful 
execution,  subtask  sequence,  rnd  a  variety 
of  other  actions. 

A  number  of  recommendations  for  the  sonar 
system  under  development  were  made  based  on 
the  findings  of  the  model  application. 


5.  Application  Components 
of  the  Process 

5.1  Test  Execution 

5.2  Data  Analysis 

5.3  Findings  Interpretation 

5.4  Conclusions  and 
Recommendations 


6.  Further  Research  Areas 

6.1  Measurement  System 
Limitations 

6.2  Research  Potentials/ 
Priorities 

*.3  Research  Planning 
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1.  State  of  the  Art  Review 
of  the  Process 

1.1  General  System 
Measurements 

1.2  System  Taxonomy 
Model  (STM) 

1.3  Overall  Conceptual 
Process  Model  (CPM) 


2.1  This  study  addressed  the  performance  of 
maintenance  personnel. 

2.2  The  maintenance  man's  mission  in  a 
man-machine  system  is  to  ensure  that  the 
machine  subsystem  is  in  prime  operating 
condition  when  the  mission  is  started. 


2.  Contextual  Components 

of  the  Process  3*1 

2.1  System  Definition 

2.2  Mission  Definition 

2.3  Environment  Definition 

2.4  General  Constraints 

2.5  Performance  3.2 

Requirements,  Ultimate 

2.6  Performance 
Criteria,  Ultimate 


A  model  battery  of  48  criterion  referenced 
Job  Task  Performance  Tests  (JTPT)  were  used 
in  this  study. 

A  scoring  scheme  for  the  measurement  of  the 
task  performance  ability  of  maintenance 
personnel  was  developed. 


3.  Analytic  Components 
of  the  Process 

3. 1  Practical  Measurable 
Attributes 

3.2  Practical  Attribute 
Measures 

3.3  Performance 
Requirements,  Specific 

3.4  Performance 
Criteria,  Specific 

3.5  Measurement  Procedures 

4.  Planning  Components 
of  the  Process 

4.1  Analytic  Methods 

4.2  Parameter  Determinations 

4.3  Apparatus  for  Testing 

4.4  Personnel  for  Testing 

4.5  Test  Plans 

5.  Application  Components 
of  the  Process 

5. 1  Test  Execution 

5.2  Data  Analysis 

5.3  Findings  Interpretation 

5.4  Conclusions  and 
Recommendations 


3.3 


3.4 


The  following  classes  of  Job  activities 
were  considered:  equipment  checkout, 
alignment/calibration,  removal/replacement , 
soldering,  use  of  test  equipment,  and 
troubleshooting. 

After  considering  product,  process  and  time 
as  to  their  appropriateness  for  scoring 
results  for  each  activity,  it  was  decided 
that  a  test  subject  had  not  reached 
criterion  until  he  had  produced  a  complete, 
satisfactory  product. 

Many  factors  were  considered  including  the 
identification  and  clarification  of  tasks 
to  be  measured,  the  hierarchical  relation¬ 
ship  of  maintenance  tasks,  the  most  effec¬ 
tive  order  of  measurement  and  the  ease  of 
test  administration. 


6.  Further  Research  Areas 

6.1  Measurement  System 
Limitations 

6.2  Research  Potentials/ 
Priorities 

6.3  Research  Planning 
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The  battery  of  tests  included  48  tests,  81 
problems  and  133  scorable  products.  A 
profile  for  displaying  th'  results  of  these 
tasks  which  attaches  meaningful  information 
to  these  numbers  wa£.  developed.  This 
profile  contained  information  regarding  a 
test  subject's  success  on  the  full  range  of 
tests  including  a  subject's  job  abilities, 
strengths,  and  weaknesses. 

Due  to  the  unavailability  of  test  subjects, 
the  tryout  was  not  as  extensive  as  planned. 
The  tryout  did  indicate  that  the  tests  as 
developed  are  administratively  feasible. 

It  is  felt  that  their  use  would  result  in 
further  modifications  and  improvements. 
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This  paper  discusses  the  status  of 
performance  measurement  for  maintenance  and 
describes  how  formal  Job  Task  Performance 
Tests  (JTPTs)  have  been  replaced  by  paper 
and  pencil  theory  and  job  knowledge  tests. 
The  paper  states  that  research  has  indicated 
that  these  latter  methods  have  proved 
unsuccessful  in  terms  of  measuring  the 
ability  to  perform  maintenance  tasks. 

The  author  describes  the  Air  Force  Human 
Resources  Laboratory  (AFHRL)  efforts  to  give 
consideration  to  the  man-machine  interface 
in  performance  measurement.  One  result  of 
this  effort  has  been  the  articulation  of  a 
structure  for  handling  maintenance  functions 
and  their  complex  relationships  in  a 
systematic  manner. 

Several  problems  concerning  the  research, 
development,  and  implementation  of 
performance  measurement  were  discussed  and 
the  paper  ended  with  a  proposal  for  future 
research  and  development  efforts  based  on 
what  has  been  accomplished.  Five  general 
areas  of  consideration  were  recommended: 

(1)  Refinement  of  Model  JTPT  Battery 
(Electronic  Maintenance). 

(2)  Refinement  of  Symbolic  Substitutes 
(Electronic  Maintenance). 

(3)  Development  of  Model  JTPT  Battery 
(Mechanical  Maintenance). 

(4)  Development  of  Symbolic  Substitutes 
(Mechanical  Maintenance). 

(5)  Job  Aptitude  Test  Research  based  on 
results  of  JPPT. 
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1.  State  of  the  Art  Review 
of  the  Process 

1.1  General  System 
Measurements 

1.2  System  Taxonomy 
Model  (STM) 

1.3  Overall  Conceptual 
Process  Model  (CPM) 

2.  Contextual  Components 
of  the  Process 

2.1  System  Definition 

2.2  Mission  Definition 

2.3  Environment  Definition 

2.4  General  Constraints 

2.5  Performance 
Requirements,  Ultimate 

2.6  Performance 
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1.1  This  report  summarizes  the  procedures 
followed  in  developing  performance  measures 
for  AIT  (Advanced  Individual  Training) 

Armor  Crewmen. 

The  objective  of  the  study  was  to  establish 
definitive  performance  measures  for  those 
requirements  considered  relevant  to  job 
entry  and  to  present  these  measures  in  the 
training  objective  format. 

3.1  The  performance  measures  to  support 
revisions  in  the  Armor  Crewmen  AIT 
program  were  developed  in  three  phases. 


3.  Analytic  Components 
of  the  Process 

3.1  Practical  Measurable 
Attributes 

3-2  Practical  Attribute 
Measures 

3.3  Performance 
Requirements,  Specific 

3.4  Performance 
Criteria,  Specific 

3-5  Measurement  Procedures 

4.  Planning  Components 
of  the  Process 

4.1  Analytic  Methods 

4.2  Parameter  Determinations 

4.3  Apparatus  for  Testing 

4.4  Personnel  for  Testing 

4.5  Test  Plans 

5.  Application  Components 
of  the  Process 

5.1  Test  Execution 

5.2  Data  Analysis 

5.3  Findings  Interpretation 

5.4  Conclusions  and 
Recommendations 


(1)  Development  of  a  tasklist. 

(2)  Written  objectives  and  tests. 

(3)  Selected  tasks  for  training. 

In  the  first  phase  a  working  committee 
(representatives  from  the  Training  Center, 
HumRRO  and  the  Armor  School)  developed  a 
task  inventory.  Following  this  activity 
each  task  was  coded  to  reflect  level  of 
mastery  that  a  man  should  achieve  prior  to 
entering  an  Armor  unit.  In  phase  2 
training  objectives  were  developed.  The 
goal  of  these  objectives  was  to  define 
entry  level  mastery  for  each  task.  The 
objectives  included  conditions  for  task 
performance  and  standards  to  evaluate 
proficiency  of  task  performance.  The  third 
phase  was  to  identify  those  tasks  that  were 
appropriate  and  feasible  for  AIT. 


6.  Further  Research  Areas 

6.1  Measurement  System 
Limitations 

6.2  Research  Potentials/ 
Priorities 

6.3  Research  Planning 
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In  instances  where  time  standards  were  not 
available  in  published  doctrine,  they  were 
recommended  by  subject  matter  experts. 

When  selecting  the  tests  to  be  used  for 
training,  they  were  submitted  to  expert 
reviewers  to  assure  accuracy  of 
checkpoints,  appropriateness  of  time  limits 
and  feasibility  of  administering  tests. 

The  majority  of  the  tasks  involved  an 
observable  procedure  and  most  of  the 
standards  are  tests  of  the  process  in 
performing  the  tasks.  A  time  limit  is 
included  on  the  assumption  that  time  to 
perform  a  task  is  an  integral  component  of 
mastery. 
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State  of  the  Art  Review  1.1 

of  the  Process 

1.1  General  System 
Measurements 

1.2  System  Taxonomy 
Model  (STM) 

1.3  Overall  Conceptual  1  o 

Process  Model  (CPM) 

Contextual  Components 
of  the  Process 

2.1  System  Definition 

2.2  Mission  Definition 

2.3  Environment  Definition 

2.4  General  Constraints 


This  report  discusses  methods  to  control 
the  variance  contributed  by  the  human 
operator  which  influence  the  total  system 
per formance. 

It  is  proposed  that  human  operators  of  the 
system  which  is  being  evaluated  be  selected 
according  to  criterion  which  eliminates 
personnel  who  fall  at  extreme  ends  of  the 
distribution  of  values  of  relevant 
characteristics. 
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Selection  criteria  should  include: 

(1)  Military  Occupational  Specialty  (MOS) , 
complete  with  skill  level  suffix. 

(2)  Physical  dimensions. 

(3)  Sensory  acuity. 

When  selection  has  taken  place,  a  more 
detailed  description  of  his/her 
characteristics  should  be  obtained  as 
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5.1  Test  Execution 
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follows: 

(1)  Actual  value  for  the  participant  on 
each  of  the  selection  criterion 
measurements. 

(2)  Scores  from  aptitude  and  MOS  tests. 

(3)  Experience  (in  months)  in  the  relevant 
MOS. 

(4)  Total  service  time. 


Other  related  training,  results  of 
tests,  level  of  performance  attained. 

6.2  Research  Potentials/ 

Priorities 

6.3  Research  Planning 


6.  Further  Research  Areas  (5) 

6.1  Measurement  System 
Limitations 
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(6)  Education. 


It  is  noted  that  this  procedure  is  not  a 
radical  change  from  current  procedures.  It 
suggests  that  some  consistency  be  imposed 
in  the  selection  process. 

5.4  It  is  felt  that  a  long-run  benefit  from  the 
proposed  approach  would  be  the  development 
of  performance  based  selection  criteria. 

If  a  data  base  is  built,  it  should  be 
possible  to  develop  a  profile  of  levels  of 
the  characteristics  which  are  required  by 
similar  tasks  in  other  systems. 
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(2)  Operational  Test  and  Evaluation 
(0TAE).  A  third  category.  Production 
Acceptance  Test  and  Evaluation  (PATAE) 
consisted  of  testing  production  items 
to  demonstrate  that  contract  require¬ 
ments  of  the  system  were  met. 

HFE  TAE  (Human  Factors  Engineering  -  Test 
and  Evaluation)  existed  to 
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(1)  Demonstrate  system,  equipment,  and 
facility  design  conformance,  to  human 
engineering  design  criteria. 

(2)  Determine  man's  contribution  to 
performance  requirements. 

(3)  Quantify  man-machine  interactive 
measures  of  system. 

(*D  Detect  undesirable  design  on 
procedural  features  of  system. 

Through  these  steps,  this  guide  was 
designed  to  match  particular  techniques  to 
particular  applications,  and  describes  how 
to  utilize  the  techniques. 

The  "real  world"  requirements  that  the  HFE 
needs  to  verify  in  order  to  optimize  the 
man-machine  interface  were  technical 
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requirements  of  human  performance,  design 
criteria,  safety,  training,  personnel 
skill/quantity,  technical  publications,  and 
life  support  criteria;  decision-making 
structures,  data  inputs,  timing,  level  of 
detail,  applications. 

T&E  techniques  consisted  of  direct  manual, 
system  measurement,  indirect  manual, 
automatic  recording,  physiological,  and 
simulation. 
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_ ABSTRACT _ 

This  report  is  a  final  summary  of  many 
individual  and  interdisciplinary  group 
studies  conducted  as  a  part  of  the  total 
project.  Each  study  had  the  goal  of  the 
project,  performance  and  recovery  of  man 
within  a  work  system,  as  a  basic  direction. 

In  many  work  oriented  groups  it  is  required 
that  missions  be  fulfilled  while  in,  or 
Just  after  being  in,  environments  such  as 
noise,  vibration,  or  unfavorable  climatic 
conditions.  All  of  those  factors  affect 
human  performance  drastically.  Other 
variables  which  affect  human  performance 
include  task  demands,  motivational  level, 
type  of  organization,  and  nutritional 
status,  as  well  as  others.  It  was  the 
purpose  of  this  research  program  to 
generate  basic  data  concerning  human 
performance  and  recovery  under  the 
conditions  mentioned  above.  Experiments 
were  conducted  to  study  the  effects  of 
these  variables  singly  and  in  combination 
on  human  performance  and  recovery. 

All  experimentation  was  conducted  within 
the  framework  of  a  task  model  as  follows: 

(1)  Items  Measured  (Dependent  Variables): 

a.  Performance  Responses  -  Latency, 
accuracy,  length  of  time, 
increments  and  decrements,  etc. 
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b.  Physiological  Responses  -  Oxygen 
consumption,  GSR  heart  rate,  skin 
and/or  core  temperature,  urine 
analysis,  etc. 

c.  Training  Responses  -  Subjective 
states  or  moods,  attitude, 
learning,  etc. 

(2)  Measurement  Conditions  (Independent 
Variables) : 

a.  Task  Demands  -  Amount  of  work, 
complexity,  difficulty,  work-rest 
cycles  (duration) ,  and  ambiguity. 

b.  Level  of  Motivation  - 
Interpersonal  response  traits, 
drive  state,  reinforcement  value, 
amount  of  deprivation, 
instructional  set,  etc. 

c.  Nutritional  History  -  Meal 
spacing,  intake,  social  class, 
ethnic  group  membership. 

d.  Environments  -  Effective 
temperature,  vibration,  normal, 
lighting  conditions,  etc. 

e.  Work  Systems  Setting  - 
Individual,  team,  crew. 

(3)  Tasks  Included: 

a.  Man  as  a  Machine 

b.  Man  as  a  Machine  User 

c.  Man  as  a  Machine  Controller 

d.  Man  as  a  Machine  Servant 

e.  Man  as  a  Social  Interactor 

In  addition  to  the  task  model,  research  was 
guided  by  the  military  interest  in 
continuous  operations.  The  research  team 
emphasized  experimentation  with  relatively 
long-term  repeated  measurement  designs. 
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1.1  The  worth  of  any  system  is  directly 

relatable  to  the  confidence  the  user  has  in 
its  ability  to  perform  its  designated 
tasks.  Systems  effectiveness  and  its 
fiscal  corollary,  cost  effectiveness, 
consititute  the  most  important  area  of 
concern  to  research  and  development 
management.  As  a  result  the  Weapon  System 
Effectiveness  Industry  Advisory  Committee 
(WSEIAC)  was  established  to  develop  a 
technique  to  apprise  management  of  current 
and  predicted  system  effectiveness  at  all 
phases  of  system  life.  The  technique  known 
as  the  WSEIAC  model  was  developed  for  this 
purpose. 
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This  report  augments  the  basis  WSEIAC  model 
development  in  three  areas.  One  of  these 
areas  is  the  partitioning  of  system 
capability  into  hardware  and  personnel 
adequacy. 

System  effectiveness  may  be  defined  as  the 
probability  that  the  man-machine  complex 
system  will  successfully  meet  an 
operational  demand  and  fulfill  the 
predetermined  mission  objective  within  a 
given  mission  time  when  operated  under 
stated  conditions. 


5.  Application  Components 
of  the  Process 

5.1  Test  Execution 

5.2  Data  Analysis 

5.3  Findings  Interpretation 

5.4  Conclusions  and 
Recommendations 


6.  Further  Research  Areas 

6.1  Measurement  System 
Limitations 

6.2  Research  Potentials/ 
Priorities 

6.3  Research  Planning 


Gex,  R.C.  Personnel  subsystem  testing  and  evaluation  for  missiles  and  space 
systems:  An  annotated  bibliography  (SB-61-21).  Sunnyvale,  CA:  Lockheed 
Aircraft  Corp.,  Missiles  and  Space  Division,  April  1961. 


1  Topics  Relevant 

1  1 

1  1 

1  to  System  Development 

(Topic! 

land  Evaluation  Technology 

1  No.  ! 

ABSTRACT 

1.  State  of  the  Art  Review 
of  the  Process 

1.1  General  System 
Measurements 

1.2  System  Taxonomy 
Model  (STM) 

1.3  Overall  Conceptual 
Process  Model  (CPM) 

2.  Contextual  Components 
of  the  Process 

2.1  System  Definition 

2.2  Mission  Definition 

2.3  Environment  Definition 

2.4  General  Constraints 

2.5  Performance 
Requirements,  Ultimate 

2.6  Performance 
Criteria,  Ultimate 


1.1  In  the  face  of  radical  restrictions  to 
traditional  psychometric  approaches, 
progress  has  been  made  in  personnel 
measurement  programs  in  connection  with 
weapon  systems  developed  by  all  three  major 
branches  of  the  military.  It  was  felt  that 
this  program  could  not  be  found  in  single 
sources  and  therefore  this  literature 
search  was  designed  to  provide  an  annotated 
bibliography  in  four  major  areas: 

(1)  Performance  evaluation  and  unit 
proficiency. 

(2)  Personnel  requirements. 
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(3)  Training  and  training  equipment. 

(4)  Human  engineering. 

Although  many  of  the  reports  are  concerned 
with  equipment  design,  training  development, 
and  generation  of  personnel  requirements 
information,  it  is  very  likely  that  the 
rationales  and  criteria  adopted  for  such 
activities  can  provide  guidelines  for  the 
subsequent  test  and  evaluation.  It  is 
believed  that  this  bibliography  will  be 
useful  to  personnel  subsystem  specialists 
in  particular. 
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This  is  a  follow-on  to  a  previous  investiga 
tion  of  panel  layout.  The  purpose  of  this 
study  is  to  determine  whether  or  not  previ¬ 
ous  findings  generalize  to  discontinuous 
action  sequences. 

The  system  under  study  is  a  panel  layout, 
controls,  display  hardware  and  operators. 

The  arrangement  or  layout  of  controls  and 
display  should  optimize  operator 
performance. 

Analyses  of  variance  were  conducted  on  the 
two  major  dependent  variables,  total  oper¬ 
ating  time  per  trial,  and  total  errors  of 
omission  and  commission  per  trial. 

The  basic  apparatus  consisted  of  four  major 
components:  a  subject's  console;  an 
experimenter's  console;  a  diode  matrix; 
recording  and  auxiliary  equipment. 

Subjects  were  200  male  college  students 
between  the  ages  of  18  and  26  years,  who 
met  certain  physical  and  scholastic 
criteria.  All  subjects  were  paid  for  their 
participation. 
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4.5  The  experimental  design  was  formed  by 

combining  factorially  three  experimental 
task  operations  with  four  principles  to 
produce  12  panels  or  groups  of  10  subjects 
each.  The  principles  applied  were: 
Sequencing  (SEQ),  Functional  Grouping  (FG), 
Location  by  Frequency  (FREQ)  or  Location  by 
Importance  (IMP). 

The  action  sequence  was  made  discontinuous 
by  dividing  the  dominant  action  path  of  the 
continuous  study  into  twelve  coherent 
segments  with  each  segment  beginning  with  a 
display  and  ending  with  a  control  action 
followed  by  a  delay  period. 

5.3  The  major  finding  of  this  study  is  that  the 
overall  superiority  of  the  Sequencing 
Principle  has  been  confirmed  using  a 
discontinuous  presentation  of  the 
operator's  task.  It  is  concluded  that  the 
most  powerful  tool  available  to  the  control 
panel  designer  is  optimum  application  of 
the  Sequencing  Technique. 
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1.3  The  paper  describes  a  visual  detection 

model  that  has  been  subjected  to  a  rather 
thorough  validation  study.  The  model 
provides  estimates  of  target  acquisition 
probability  as  a  function  of  range.  These 
estimates  are  used  as  a  major  input  to  an 
overall  system  effectiveness  model.  The 
visual  detection  model  is  a  "glimpse 
model",  i.e.,  the  cumulative  probability  of 
target  acquisition  is  obtained  as  the 
product  of  a  series  of  single  glimpse 
probabilities. 

3.5  Validation  of  the  model  was  obtained  by 
comparing  the  model's  predictions  for  34 
selected  targets  with  the  results  obtained 
in  an  experiment  in  which  a  number  of 
observers  viewed  a  motion  picture  presenta¬ 
tion  of  a  flight  over  those  targets.  The 
overall  product-moment  correlation  between 
predicted  and  experimental  median  ranges 
was  +.53. 

4.2  Certain  "unfinished  business"  problems 

remain  with  each  of  the  parameters  involved 
in  the  model.  These  parameters  are: 


4.1 

Analytic  Methods 

(1) 

4.2 

Parameter  Determinations 

4.3 

Apparatus  for  Testing 

4.4 

Personnel  for  Testing 

4.5 

Test  Plans 

(2) 

Application  Components 

of  the  Process 

(3> 

5.1 

Test  Execution 

5.2 

Data  Analysis 

(4) 

5.3 

Findings  Interpretation 

5.4 

Conclusions  and 

Recommendations 


Further  Research  Areas 

6.1  Measurement  System 
Limitations 

6.2  Research  Potentials/ 
Priorities 

6.3  Research  Planning 


The  probability  of  looking  at  the 
target. 

Sensor  resolution. 

Inherent  object/background  contrast. 
Sky/ground  luminance  ratio. 
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(5)  Aircraft  altitude  above  the  target. 

(6)  "Visibility"  or  meteorological  range. 

(7)  Threshold  contrast. 

(8)  Target  dimensions. 

(9)  Number  of  required  resolution 
elements . 

6.1  A  number  of  parameters  currently  are 
"missing”  from  the  model,  including: 

(1)  Level  of  confidence  [of  the  observer] 

(2)  Briefing  and  reference  materials. 

(3)  Search  techniques. 


At  present  it  is  not  known  how  these 
parameters  should  be  handled. 
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The  objectives  of  this  study  were  to  assess 
the  effectiveness  of  a  performance  measure¬ 
ment  package  and  to  evaluate  the  efficiency 
of  operational  test  and  evaluation  proce¬ 
dures  developed  as  a  data  gathering  tool 
for  performance  measurement.  An  operational 
test  and  evaluation  was  conducted  on  the 
capabilities  of  the  Automated  Adaptive 
Flight  Training  System  (AAFTS)  system  prior 
to  this  study.  The  data  gathered  during 
the  course  of  that  effort  provided  the  data 
source  for  this  evaluation. 

The  AAFTS  was  installed  on  the  F-4E  #15 
Weapon  System  Training  Set  (WSTS).  The 
subject  of  this  evaluation  was  the  aircrew 
skills  in  Air-to-Air  Intercepts  (AAI) . 

The  AAI  program  is  broken  into  nine  modes: 


(1) 

Single  turn  attacks 

(2) 

Increasing/decreasing  aspect 

attacks 

(3) 

Stern  conversion  attacks 

(4) 

Stern  conversion  ID  passes 

(5) 

Snap-ups 

(6) 

Single  turn  attacks  with  stern 
conversion  reattacks 

(7) 

No  lock-on  attacks 

(8) 

No  Ground  Control  Intercept 
attacks 

(GCI ) 
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(9)  Tactical  intercepts 

Each  of  the  above  modes  has  a  number  of 
steps  within  it.  The  steps  were  designed 
*0  present  progressively  more  difficult 
intercepts  as  a  function  of  the  values  of 
certain  variables. 

3.1  In  the  AAI  environment,  variables  often 
measured  the  geometry  of  attack,  missile 
launch  parameters,  and  other  variables 
deemed  important  to  an  intercept.  The 
AAFTS  device  used  a  scoring  system  of  this 
nature.  A  set  of  28  dependent  variables 
were  determined  by  the  Operational  Training 
Development  Team  (OTDT).  These  variables 
included  measures  of  how  closely  the  crews 
stayed  to  assigned  airspeed;  altitudes  and 
headings;  how  accurately  the  crew  launched 
missiles;  whether  or  not  correct  missile 
launching  control  procedures  were  followed/ 
accomplished;  and  other  variables  specific 
to  particular  types  of  AAIs. 

3.2  All  variables  had  acceptable  bandwidths  of 
performance  that  the  crew  had  to  maintain 
in  order  to  accumulate  scoring  points. 

While  the  AAFTS  has  a  28  variable  standard 
format,  only  these  variables  relevant  to 
the  particular  engagement  underway  were 
scored  by  the  AAFTS  system. 

4.1  T-statistics  were  used  to  test  differences 
between  the  total  scores  (a  summation  of 
all  the  variables  measured  for  each 
engagement  completed)  achieved  by  the  crews 
for  each  mode.  They  were  also  used  to  test 
differences  between  two  types  of  crews  and 
on  a  variable-by-variable  basis  across 
modes  wherever  the  particular  variable  was 
measured.  The  AAFTS  device  had  specific 
point  values  assigned  to  performance  error 
tolerance  limits  for  each  variable.  An 
equal  interval  scoring  technique  was  also 
employed  to  more  accurately  satisfy 
assumptions  on  which  the  statistic  is  based 
(i.e.,  a  variable  with  scored  values  of  10, 
8,  6,  3.  0  was  arbitrarily  rescaled  to  4, 
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3,  2,  1,  0)  and  then  between-groups 
differences  were  examined  by  mode  and  by 
variable. 

4.3  The  AAFTS  was  a  parasitic  computer-based 
instructional  device  interfaced  with  the 
F-4E  WSTS  #15.  The  system  can  auto¬ 
matically  score  advancing  aircrews  through 
various  programs  including  a  116-step  AAI 
program. 

4.4  Seven  operationally  qualified  crews  and 
five  student  crews  participated  in  the 
study. 

4.5  Training  received  by  both  crews  was 
constrained  by  Tactical  Air  Command 
training  requirements  and  OT&E  limitations. 
The  operational  crews  received  experience 
in  flying  the  F-4B  WSTS  #15  and  the  student 
crews  were  instructed  using  the  AAFTS 
training  device.  Crews  flew  and  were  scored 
on  nine  different  types  of  air-to-air 
intercepts  that  were  programmed  into  the 
AAFTS  device.  The  crews  flew  the  simulator 
under  different  conditions — the  operational 
crews  were  adaptively  scheduled  by  the 
AAFTS  as  often  as  possible  while  the 
student  crews  flew  what  was  dictated  by  the 
instructor. 

5.3  The  data  revealed  significant  differences 
favoring  the  operational  crews  in  two  types 
of  attacks,  single  turns  and  stern 
conversions.  On  a  variable-by-variable 
basis,  there  were  three  significant 
differences  between  operational  and  student 
crews,  two  of  which  indicated  superior 
performance  by  student  crews. 

5.4  The  data  warranted  the  following 
conclusions: 

(1)  The  AAFTS  as  an  instructional  tool  has 
the  potential  to  provide  standardized 
and  objectively  scored  training/ 
evaluation  scenerios  for  aircrews. 
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(2)  The  performance  measurement  package 
performed  adequately  as  an 
informational  feedback  tool  but 
requires  more  research  to  select  and 
validate  variables  that  discriminate 
better  between  aircrew  skill  levels. 
Failure  to  discriminate  could  have 
occurred  because  the  variables  and 
their  weights  were  based  on  the 
opinions  of  subject  matter  experts  and 
not  on  empirical  evidence. 

(3)  Use  of  the  OT&E  procedures  was 
minimally  effective  for  a  performance 
measurement  but  served  well  as  a  means 
for  acquiring  aircrew  subjective 
impressions  on  the  systems'  overall 
traiuing/evaluation  potential. 
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1 . 1  General  System 
Measurements 

1.2  System  Taxonomy 
Model  (STM) 

1.3  Overall  Conceptual 
Process  Model  (CPM) 


1.3  The  purpose  of  this  paper  is  to  describe 
some  of  the  deficiencies  in  Air  Force 
proficiency  assessment  and  to  indicate 
broadly  how  these  deficiencies  can  be 
corrected.  In  addition,  suggestions  are 
given  for  promising  immediate  research  and 
development  efforts. 


2.  Contextual  Components 
of  the  Process 

2.1  System  Definition 

2.2  Mission  Definition 

2.3  Environment  Definition 

2.4  General  Constraints 

2.5  Performance 
Requirements,  Ultimate 

2.6  Performance 
Criteria,  Ultimate 


2.1  Air  Force  maintenance  personnel  and  their 
training  was  the  system  addressed. 

2.2  The  training  must  accomplish  the  job  of 
making  the  maximum  number  of  personnel 
adequately  proficient  during  the  training 
period. 


3.  Analytic  Components 

of  the  Process 

3.1  Practical  Measurable 
Attributes 

3.2  Practical  Attribute 
Measures 

3.3  Performance 
Requirements,  Specific 

3.4  Performance 
Criteria,  Specific 

3.5  Measurement  Procedures 

4.  Planning  Components 

of  the  Process 

4.1  Analytic  Methods 

4.2  Parameter  Determinations 

4.3  Apparatus  for  Testing 

4.4  Personnel  for  Testing 

4.5  Test  Plans 

5.  Application  Components 

of  the  Process 

5.1  Test  Execution 

5.2  Data  Analysis 

5.3  Findings  Interpretation 

5.4  Conclusions  and 
Recommendations 

6.  Further  Research  Areas 

6.1  Measure^en*  System 
Limitations 

6.2  Research  Potentials/ 
Priorities 

6.3  Research  Placing 


The  Air  Force  competes  with  industry  for 
the  services  of  trained  personnel  and  is 
not  usually  able  to  retain  its  personnel 
longer  than  a  four-year  tour.  There  is  a 
need  to  train  the  maintenance  personnel  at 
a  great  variety  of  tasks,  some  of  which 
require  high  levels  of  skill  in  a  short 
period  of  time  without  further  education, 
cross-training,  or  re-training. 

The  researchers  showed  that  there  are  no 
suitable  over-all  objective  measures  of 
performance  and  there  was  no  ultimate 
criteria  against  which  training  standards 
and  intermediate  measures  of  achievement 
can  be  evaluated. 

It  was  concluded  that  the  many  deficiencies 
in  current  proficiency  measurement  of 
maintenance  personnel  can  be  corrected 
through  a  vigorous  program  of  development 
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and  Implementation.  However , _such  a 
program  must  have,  as  an  integral  aspect, 
the  definition  of  training  and  job 
performance  standards  for  each  Air  Force 
maintenance  position.  Only  whtn 
performance  is  assessed  against  such 
standards  can  all  of  the  requirements  for 
proficiency  measurement  be  met. 

It  is  recommended  that  the  definition  of 
training  and  job  performance  standards  for 
selected  critical  maintenance  positions  be 
undertaken  in  association  with  the 
development  of  performance  measures  in 
order  to  serve  as  a  model  which  can  be 
applied  to  other  Air  Force  maintenance 
positions. 

6.2  Recommendations  were  made  for  future 
research.  They  included: 

(1)  Research  to  refine  principles  and 
techniques  for  assessing  individual 
performance  in  trouble:  -ooting  and 
other  complex  tasks. 

(2)  Research  be  sustained  on  improving 
maintenance  records  and  supervisers' 
ratings  as  criteria  of  performance  on 
the  job. 

(3)  A  practical,  procedural  handbook  be 
developed  which  can  be  followed  in 
assessing  performance  capabilities. 

(4)  That  the  preceding  efforts  be  aimed  at 
specifications  for  the  development  of 
proficiency  measures  for  inclusion  in 
weapon  system  development  contracts. 
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1.3  This  article  dealt  with  exposure  of 
categories  of  driver-vehicles  to  motor 
vehicle  traffic  (exposure  in  the  narrow 
sense)  or  to  all  external  hazards  (exposure 
ir>  the  wide  sense) . 

3.5  A  number  of  different  models  for  the 

measurement  of  exposure  were  considered. 

5.3  The  Koornstra  method  showed  sufficient 
agreement  with  real  data  to  justify  further 
exploration. 

5.4  The  next  step  would  be  validation  through 
use  of  other  categorization,  other  time 
periods  and  other  locations  than  used. 


4.  Planning  Components 
of  the  Process 

4.1  Analytic  Methods 

4.2  Parameter  Determinations 

4.3  Apparatus  for  Testing 

4.4  Personnel  for  Testing 

4.5  Test  Plans 


5.  Application  Components 
of  the  Process 

5.1  Test  Execution 

5.2  Data  Analysis 

5.3  Findings  Interpretation 

5.4  Conclusions  and 
Recommendati  ons 

6.  Further  Research  Areas 

6.1  Measurement  System 
Limitations 

6.2  Research  Potentials/ 
Priorities 

6.3  Research  Planning 
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Criteria,  Ultimate 

Analytic  Components 
of  the  Process 

3.1  Practical  Measurable 
Attributes 

3.2  Practical  Attribute 
Measures 

3.3  Performance 
Requirements,  Specific 

3.1)  Performance 

Criteria,  Specific 
3.5  Measurement  Procedures 

Planning  Components 

of  the  Process 

A.1  Analytic  Methods 

A.  2  Parameter  Determinations 

A.  3  Apparatus  for  Testing 

)).))  Personnel  for  Testing 

A. 5  Test  Plans 
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5.1  Test  Execution 
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5.4  Conclusions  and 
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In  this  report  an  adaptive  model  of  the 
development  process  in  weapons  systems  is 
presented  which  enables  decision-makers  to 
determine  which  tests,  if  any,  should  be 
performed  at  a  given  stage  and  what 
corrective  actions,  if  any,  should  be  taken 
when  test  results  are  known. 

The  final  checkout  p!:ase  of  the  system  is 
considered  first.  A  model  developed  for 
the  purpose  of  determining  the  optimal 
troubleshooting  procedure  when  an  operating 
system  fails  is  an  appropriate  abstraction 
of  this  phase.  The  model  is  then 
generalized  to  include  the  case  in  which 
the  object  tested  is  destroyed.  In 
addition,  sufficient  conditions  for  the 
optimality  of  the  test  procedures  are 
given. 

Next,  the  problem  of  test  selection  in 
early  stages  of  development  is  superimposed 
on  the  final  checkout  phase.  A  dynamic 
programming  formulation  is  given  which 
enables  one  to  determine  which  test,  if 
any,  should  be  performed  at  any  given  stage 
and  what  corrective  action,  if  any,  should 
be  taken.  As  a  by-product,  means  for 
determining  net  benefits  of  a  given  test  at 
a  given  stage  is  obtained. 

The  situation  in  which  there  is  serial 
dependence  among  certain  test  information 
messages  is  examined.  Finally,  it  is  shown 
that  a  mooel  of  parallel  research  and 
development  efforts  may  be  viewed  as  a 
special  case  of  the  model  developed  in  this 
paper. 
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5.M  While  this  model  is  presented  in  terms  of 
relatively  simple  systems  and  tests,  it  is 
capable  of  handling  systems  and  tests  of  a 
highly  complex  nature. 
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This  report  justifies  the  use  of 
established  statistical  methods  to  obtain 
confidence  intervals  for  measures  of  weapon 
system  effectiveness. 

These  methods  apply  to  effectiveness 
measures  that  are  generated  by  sampling 
mathematically  simulated  weapon/target 
encounters. 

There  are  two  general  sources  of 
unreliability  in  computer  simulations.  (1) 
the  computer  program  does  not  model  the 
real  world  as  closely  as  it  should  and  (2) 
the  number  of  encounters  simulated  is 
insufficient  to  yield  statistically 
significant  conclusions.  The  first  sources 
of  unreliability  can  be  revealed  and 
remedied  by  a  model  validation  procedure. 
The  second  source  of  unreliability  can  be 
reduced  by  standard  statistical  methods  by 
determining  the  proper  sample  size. 

It  is  demonstrated  in  this  report  that 
relatively  straightforward  statistical 
techniques  can  be  used  to  generate 
confidence  limits.  Methods  applicable  for 
use  with  continuously  distributed  functions 
and  with  zero-or-one  measures  of 
effectiveness  are  briefly  described  with 
detailed  illustrative  examples. 


6.  Further  Research  Areas 

6.1  Measurement  System 
Limitations 

6.2  Research  Potentials/ 
Priorities 

6.3  Research  Planning 
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1 . 1  This  report  is  concerned  with  human  factors 
engineering  methodologies.  It  is 
particularly  concerned  with  quantitative 
methods  applicable  to  system  engineering 
efforts. 

The  role  of  human  factors  engineering  in 
each  phase  of  system  development  is 
summarized.  The  methods  used  by  human 
factors  personnel  in  performing  the 
activities  are  largely  heuristic.  They 
include  both  quantitative  and 
non-quantitative  techniques.  The  variety 
of  non-quantitative  techniques  is 
illustrated  in  the  following  table. 
Quantitative  techniques  come  from  both  the 
physical  and  behavioral  sciences. 


14.  Planning  Components 
of  the  Process 
*1.1  Analytic  Methods 
4|.2  Parameter  Determinations 
K.3  Apparatus  for  "'esting 
4|.*4  Personnel  for  Testing 
<4.5  Test  Plans 


5.  Application  Components 
of  the  Process 

5.1  Test  Execution 

5.2  Data  Analysis 

5.3  Findings  Interpretation 
5. *4  Conclusions  and 

Recommendations 
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Illustrative  Non-Quantitative  Techniques  in  Human  Factors  Engineering 

NAPRATIVE  TABULAR  GRAPHIC 

Position  Description  Task  Equipment  Analysis  Operational  Sequence 

Diagram 

Job  Description  Task  Analysis  of  Maintenance  Sequence 

Procedures  Diagram 

Position-Equipment-Task  Time  Line  Charts 

Summary 

Task  Analysis  Worksheet  System  Analysis  and 

Integration  Technique 


Equipment  Evaluation  System  Requirements  and  Integration  Task  Index 
Guide  Capability  Analysis 

Human  Engineering  Maintenance  and  Handling  Mission  Profiles 

Checklist  Analysis 

Physical  Demands  Form  Operational  and  Support  Link  Analysis 

Data  Collection  Sheet 

Job  Psychograph  Performance  Observation  Function  Flow  Diagram 

Record  Form 

Work  Characteristic  Form  Performance  Factor  Form  Multiple  Process  Chart 

Man-Machine  Chart 

Task  Analysis  for  Error  Information-Decision- 
Identification  Action  Chart 


Task  Analysis 
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In  defining  an  approach  to  the  problems  of 
incorporating  human  factors  into  system 
effectiveness  modeling  three  basic  issues 
should  be  considered: 

(1)  The  ways  in  which  models  are  used 

(2)  The  kinds  of  human  engineering  data 
that  are  available, 

(3)  The  ways  in  which  people  are  used  in 
systems . 


A  representative  sample  of  some  of  the  more 
important  types  of  models  are  discussed. 
These  include: 

(1)  On-line  simulations. 

(2)  Control  theory  models. 

(3)  Process  models. 

(H)  Task  analysis  models. 

(5)  Human  reliability  models. 

(6)  Operability  index. 

(7)  Technique  for  human-error  rate 
prediction  (THERP). 

(8)  Technique  for  establishing  personnel 
performance  standards  (TEPPS) . 

(9)  "Logical  man". 


(10)  Special  models 
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2.1  System  Definition  _  . 
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2.3  Environment  Definition 

2. H  General  Constraints 

2.5  Performance 
Requirements,  Ultimate 

2.6  Performance  3-2 

Criteria,  Ultimate 

Analytic  Components 
of  the  Process 

3.1  Practical  Measurable 
Attributes 

3.2  Practical  Attribute 
Measures 

3.3  Performance 
Requirements,  Specific 
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Crlteria,  Specific 

3.5  Measurement  Procedures 

Planning  Components 
of  the  Process 

4.1  Analytic  Methods 

•l .2  Parameter  Determinations  i|,1 

•1.3  Apparatus  for  Testing 
*1.4  Personnel  for  Testing 

4.5  Test  Plans 

Application  Components 
of  the  Process 

5.1  Test  Execution 

5.2  Data  Analysis 

5.3  Findings  Interpretation 

5.4  Conclusions  and 
Recommendations 


The  purpose  of  this  study  was  to  explore 
the  utility,  from  a  psychometric  and  cost 
effectiveness  standpoint,  of  a  computerized 
adaptive  measurement  system  in  the  Air 
Force  technical  training  environment.  This 
phase  of  the  study  was  designed  to  produce 
an  operational  system  ready  to  test 
technical  training  students  adaptively. 

For  the  two  blocks  of  instruction,  a  task 
analysis  was  performed  and  appropriate 
measurement  items  selected. 

A  review  of  the  literature  indicated  that 
two  testing  techniques  showed  promise: 
flexilevel  testing  and  hierarchical 
testing.  These  procedures  were  modified  by 
adopting  a  two-stage  approach  whereby  a 
student  would  be  branded  into  the  testing 
net  according  to  a  regression  estimate  of 
his  predicted  score. 

Two  courses  were  selected  to  implement 
these  procedures:  one  was  selected  for 
implementation  of  the  hierarchical  testing 
and  the  other  for  flexilevel  testing. 

The  selected  measurement  items  were 
incorporated  into  a  computer  program  for 
adaptive  testing.  The  testing  procedures 
were  programed  in  the  TUTOR  language 
supported  by  the  PLATO  system  at  the 
University  of  Illinois. 
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5.M  Preliminary  conclusions  indicated  that 

adaptive  testing  offers  the  potential  for 
time  savings  up  to  50%.  It  was  found  that  a 
feasible  computer  system  to  drive  the 
adaptive  testing  strategies  could  be 
relatively  easily  developed.  The  file 
handling  and  report  generating  capabilities 
of  the  PLATO  system  were  found  to  require 
considerable  ingenuity  in  programming. 

6.2  Three  studies  were  designed  to  evaluate  the 
adaptive  testing  approach:  (a)  a  study  to 
test  and  validate  flexilevel  testing,  (b)  a 
study  to  test  and  evaluate  hierarchical 
testing,  and  (c)  a  study  to  explore  testing 
of  the  examinee  in  the  criterion  zone. 
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2.  Contextual  Components 

of  the  Process 

2. 1  System  Definition 

2.2  Mission  Definition 

2.3  Environment  Definition 

2.4  General  Constraints 

2.5  Performance 
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2.6  Performance 
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3.1  Practical  Measurable 
Attributes 
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3.5  Measurement  Procedures 


4.  Planning  Components 
of  the  Process 

4.1  Analytic  Methods 

4.2  Parameter  Determinations 

4.3  Apparatus  for  Testing 

4.4  Personnel  for  Testing 

4.5  Test  Plans 


5.  Application  Components 
of  the  Process 
5.1  Test  Execution 


In  this  report,  means  to  empirically  assess 
a  computerized  adaptive  testing  model  in  an 
ongoing  technical  training  system  were 
presented. 

The  system  studied  was  an  Inventory 
Management/Materiel  Facilities  Training 
Course  for  Air  Force  personnel. 

The  achievements  of  students  in  one  section 
of  the  course  were  measured. 

An  adaptive  testing  model  was  utilized. 

This  model  was  a  modification  of  Lord's 
(1971)  flexilevel  paradigm  and  consisted  of 
(a)  the  sequencing  of  test  items  in  a 
difficulty  hierarchy,  (b)  adaptive  entry 
into  the  test  by  students  at  a  difficulty 
level  appropriate  to  their  predicted  score, 
and  (c)  systematic  movements  of  students  up 
and  down  the  hierarchy  based  upon  their 
performance  on  preceding  items. 

Four  hundred  and  forty  four  airmen 
participated  in  this  study.  The  student 
population  selected  was  considered  fairly 
homogeneous  in  characteristics  pertaining 
to  age,  educational  background,  career 
goals,  and  military  experience.  The 
typical  student  was  male  (759  m,  259  f)  an 
average  age  of  20,  high  school  graduate 
with  less  than  one  year  Air  Force 
experience. 
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4.5  Preparation  activities  included  meetings 
with  coarse  instructors  for  briefings. 
Students  were  administered  reading  aptitude 
tests  to  provide  predictor  performance 
scores.  Test  items  were  administered 
separately  with  rate  of  presentation 
determined  by  the  student. 


5.3  The  results  of  this  testing  procedure 
revealed  a  high  positive  part-whole 
correlation  between  flexilevel  and  total 
test  scores.  Internal  consistency  indices 
were  essentially  equivalent.  The 
flexilevel  test  required  nine  fewer  items 
than  the  entire  teat,  yielding  a  length 
reduction  of  39.511  and  time  savings  of  over 
18%.  The  interpretation  of  findings 
stresses  the  potential  benefits  of  adaptive 
testing  in  terms  of  significant  time 
reductions  and  maintenance  of  high 
standards  of  test  validity. 
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2.3  Environment  Definition 

2.4  General  Constraints 

2.5  Performance 
Requirements,  Ultimate 

2.6  Performance 
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1 .2 
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3.1  Practical  Measurable 
Attributes 

3.2  Practical  Attribute 
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4.  Planning  Components 
of  the  Process 

4.1  Analytic  Methods 

4.2  Parameter  Determinations 

4.3  Apparatus  for  Testing 

4.4  Personnel  for  Testing 

4.5  Test  Plans 

1.3 

5.  Application  Components 
of  the  Process 

5.1  Test  Execution 

5.2  Data  Analysis 

5.3  Findings  Interpretation 

5.4  Conclusions  and 
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The  effectiveness  of  a  system  depends  on 
its  mission.  With  a  narrow  system  and 
mission  definition  it  Is  possible  to 
measure  the  system's  ability  to  achieve 
that  mission  directly  through  analytical 
models  or  experimental  testing.  A  broader 
definition  reduces  the  problem  of 
sub-optimization  created  by  the  narrow 
definitions;  however,  valid  analytical 
expression  is  now  threatened.  Therefore, 
the  basic  consideration  in  a  system 
definition  is  "optimization"  of  the 
measurability  -  sub-optimization 
relationship. 

The  lowest  level  of  measurable  characteris¬ 
tics  are  those  numerous  and  distinct 
attributes  of  a  system  contributing  to  its 
effectiveness,  for  example,  speed  and 
burntime.  The  second  level  set  of 
characteristics  represents  a  manner  of 
describing  the  system's  effectiveness 
through  a  smaller  and  more  general  set  of 
parameters.  The  elements  of  this  higher 
level  are  combinations  of  basic  attributes 
that  describe  a  slightly  more  general 
system  characteristic,  for  example,  the 
range  of  the  missile  (as  a  function  of  the 
above  two  lower  level  attributes) . 

From  the  definition  of  a  system  to  a 
quantitative  criterion  of  its  value,  the 
following  steps  are  identified: 
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(1)  Define  the  mission  as  broadly  as 
possible,  being  consistent  with  some 
concept  of  how  its  ability  to  achiev  ? 
the  former  can  be  expressed 
quantitatively. 

(2)  The  system  designed  to  accomplish  the 
mission  should  be  explicitly  defined 
to  some  "boundary."  The  latter  must 
separate  the  system  from  its 
environment;  contributions  from  other 
elements  or  systems  are  incidental. 

(3)  A  criterion  for  judging  the  value  of  a 
system  must  be  formulated  and/or, 

(4)  A  method  of  optimizing  the  design  or 
choice  of  system  devised. 

(5)  Based  on  the  method  of  optimization 
chosen,  certain  types  of  measurements 
must  be  obtained  for  a  complete  set  of 
characteristics  at  the  highest  level 
possible. 

(6)  A  method  of  expressing  the 
effectiveness  of  a  system  as  a 
function  of  the  elements  in  a  set  must 
be  designed,  or  if  this  measure  ca.^'t 
be  obtained  with  the  desired 
confidence  of  correctness  then, 

(7)  The  mission  should  be  redefined  such 
that  effectiveness  can  be  more 
confidently  expressed. 
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This  project  was  designed  to  provide 
substantive  information  on  the  effects  of 
government  services  in  the  community. 

Nine  service  areas  were  chosen  for 
measurement.  These  were  crime  control, 
recreation,  library,  fire  protection, 
transportation,  solid  waste,  water  supply, 
waste  water,  and  citizens'  complaints. 

For  each  of  the  above  service  areas,  the 
project  staff  developed  15  to  20  new 
effectiveness  measures. 

Examples  of  these  effectiveness  measures 
were:  the  clearance  rate  of  arrests 
through  court  system,  the  percentage  of 
individuals,  by  age  groups,  using  a  city 
recreation  facility,  circulation  per  capita 
of  library  materials,  number  and  rate  of 
injuries  and  deaths  due  to  fire,  and 
subjective  measurements  of  passenger 
comfort  on  local  transportation.  Other 
measures  included  inspection  results  and 
response  times.  No  procedures  are 
described  for  deriving  these  effectiveness 
measures. 

Two  cities  (St.  Petersburg,  FL  and 
Nashville,  TN)  were  chosen  as  the 
experimental  sites  for  this  study.  Data 
were  collected  by  appropriately  designed 
surveys  of  the  user  or  providers  of  these 
services.  A  computer  program  for 
organizing  and  analyzing  the  data  was 
developed. 
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5.4  It  was  concluded  that  this  study  provided 

the  tools  and  an  implementation  system  that 
would  afford  local  officials  significantly 
improved  information  on  their  services. 
Specifically,  the  measures  were  intended  to 
help  identify  current  problem  areas, 
indicate  trends,  inequities  in  service 
areas,  and  provide  a  data  base  for  use  in 
comparison  research  between  cities.  Cities 
throughout  the  nation  are  adopting  the 
techniques  derived  from  this  project  and 
the  techniques  are  readily  modifiable  for 


almost  any  local  government. 
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1.3  Engineering  design  deficiencies  of  two 
aircraft  were  examined  to  determine 
deficiencies  of  a  human  engineering  nature. 

3.1  The  following  human  engineering 

deficiencies  were  reported:  erroneous, 
ill-placed,  inaccessible,  inconsistent, 
incompatible,  missing,  too  complex, 
unclear,  unneeded,  and  unreadable.  In 
order  to  classify  these  deficiencies  in 
terms  of  system  hardware  components,  the 
following  categories  were  adopted: 
display,  control,  workspace,  lighting,  life 
support,  escape  egress,  seats,  parachutes, 
canopy,  and  weapons  system. 
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The  human  engineering  was  considered 
deficient  when  it  specifically  violated  the 
large  list  of  display  design  principles,  or 
the  list  of  control  design  principles,  which 
might  lead  to  display  or  control  induced 
operator  errors. 

The  critical  incident  technique  was  used  to 
catalogue,  describe  and  analyze  the  human 
engineering  design  deficiencies. 

The  results  indicate  that  there  were 
substantial  numbers  of  human  engineering 
design  deficiencies  in  relation  to  total 
engineering  deficiencies.  The  results 
suggest  that,  although  the  airframes  of  the 
two  aircraft  are  dissimilar,  there  was  a 
basic  similarity  in  the  number  and  types  of 
human  engineering  design  deficiencies. 
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The  purpose  of  this  report  is  to  describe 
and  present  a  new  method  developed  to 
evaluate  the  operational  functions 
performed  by  crewmembers. 

This  study  addressed  the  human  factors 
aspects  of  aircraft  man-machine  systems,  in 
particular  the  S-3A  aircraft  and  crew. 

A  series  of  investigations  were  conducted 
analyzing  the  operational  functions  of 
crewmembers.  Roles,  duties  and  tasks 
performed  by  each  crewmember  were 
determined.  Crewmember  judgments  were 
compiled  to  establish  relative  importance 
to  mission  success  of  these  roles,  duties 
and  tasks,  in  addition  to  the  frequency  of 
performance  on  a  typical  mission,  the 
training  necessary  to  insure  effective 
performance,  and  how  effective  the  system 
was  in  accomplishing  these  operational 
functions. 

The  technique  described  above  results  in  a 
Functional  Description  Inventory  CFDI). 

The  FDI  has  the  potential  as  a  tool  for 
providing  quantitative  assessment  of  the 
human  factors  aspects  of  aircraft 
man-machine  systems. 

It  is  recommended  that  the  development  of 
the  FDI  technique  be  validated  with  fleet 
training  crews,  etc.,  through  comparative 
analysis  of  Board  of  Inspection  Survey 
(BIS)  Yellow  Sheet  deficiencies. 


6.  Further  Research  Areas 

6.1  Measureaent  System 
Limitations 

6.2  Research  Potentials/ 
Priorities 

6.3  Research  Planning 
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The  performance  of  the  system  In  terms  of 
safety  could  be  determined  by  the  number  of 
accidents  In  which  a  driver  Is  Involved 
compared  to  test  scores. 

The  objective  of  the  study  was  to  Identify, 
through  analytic  techniques,  the  basic 
visual  and  auditory  requirements  of  the 
commercial  carrier  driver.  A  review  of  the 
literature  was  conducted,  and  a  detailed, 
systematic  examination  of  the  driving  task 
was  undertaken,  including  interviews  with 
drivers  and  direct  observation  of  the 
driving  task.  Inputs  from  these  sources 
determined  the  importance  of  each  visual 
function,  and  determined  the  auditory 
standards  to  be  tested. 
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A  device  was  designed  and  constructed  that 
provided  the  capability  of  testing 
performance  on  visual  performance 
parameters.  Basic  hearing  loss  data  was 
obtained  by  means  of  a  standard  audiometric 
technique. 

Tests  were  administered  to  a  sample  of 
commercial  carrier  drivers  whose 
performance  on  the  various  tests  was 
compared  against  past  accident  history. 
Multiple  regression  analyses  and  a 
graphical  analysis  technique  were  used  in 
data  analysis. 
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4.4  236  subjects  were  recruited  from  a  large 
trucking  firm  and  two  large  bus  companies 
also  participated  in  this  study. 

4.5  To  evaluate  experimentally  both  the  results 
of  the  analytical  effort  and  the  test 
device,  the  entire  battery  of  visual  and 
auditory  tests  were  administered  to  the 
subjects  along  with  a  questionnaire  to 
derive  biographical  and  driving  pattern 
data.  Driving  record  information  was 
obtained  from  company  files  for  each  driver 
tested,  including  total  number  of  accidents 
on  file,  number  of  "responsible"  accidents 
on  file,  number  of  months  covered  by  the 
files  and  total  number  of  accidents  and 
"responsible"  accidents  for  the  last  36 
months. 

5.3  The  new  measures  of  visual  performance  were 
shown  experimentally  to  relate  to  past 
driving  records.  However  the  limited  size 
of  the  sample  prevents  generalization  to 
the  entire  population.  It  is  recommended 
that  additional  data  be  collected  to 
increase  sample  size.  If  findings  can  be 
verified  the  probability  is  high  that  new 
visual  qualification  standards  can  be 
generated,  resulting  in  a  more  valid 
screening  procedure.  The  results  of  the 
analysis  of  auditory  data  are  not  so 
positive.  Assuming  the  results  are  not 
antifactual,  hearing  loss  standards 
presently  allowed  are  not  related  to 
accident  involvement.  The  question  of 
whether  standards  are  too  strict  cannot  be 
answered  experimentally. 
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The  purpose  of  this  research  was  to  develop 
a  standardized  methodology  for  conducting 
human  factors  evaluation  of  trucks  and 
similar  vehicles  in  operational  field 
tests. 

The  system  is  composed  of  trucks  and 
similtr  vehicles  within  the  context  of 
operational  field  tests. 

Eighty-five  human  factor  characteristics 
relevant  to  vehicle  design  and  evaluation 
were  selected  for  measurement  and  they  were 
organized  into  six  principal  categories: 
Driver  compartment,  visibility,  control  and 
control  operations,  instruments,  handling 
characteristics,  and  ride  characteristics. 

The  method  for  measurement  of  these 
characteristics  focused  on  the  assessment 
of  users'  judgements  of  the  vehicles  being 
evaluated . 

The  methodology  used  was  the  Human  Factor's 
Vehicle  Evaluation  Instrument  (HFVEI) . 

Upon  completion  of  the  test  drive,  the 
driver  was  interviewed  and  used  a  six  point 
rating  scale  -  extremely  acceptable  to 
extremely  unacceptable.  In  addition,  the 
drivers  were  required  to  rate  the  relative 
importance  (weight)  of  each  of  the  85 
characteristics. 

The  data  were  analyzed  using  Analysis  of 
Variance  (ANOVA)  and  post-hoc  multiple 
comparison  techniques. 
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4.2  Test  parameters  included  the  elimination  of 
those  drivers  who  could  not  meet  minimum 
performance  standards;  the  training  of 
interviewers,  the  familiarization  of 
drivers  to  procedure;  the  establishment  of 
a  test  course  with  a  variety  of  representa¬ 
tional  tasks  over  terrain  appropriate  for 
vehicles.  The  order  of  driving  the  course 
was  counterbalanced  and  the  driver  inter¬ 
viewed  while  still  in  the  driving  seat. 

4.3  This  evaluation  compared  a  non-standard 
3-1/2-ton  cargo  truck  with  both  a  standard 
U.S.  Army  2-1/2-ton  truck  and  a  5-ton  cargo 
truck.  The  test  took  place  around  a 
four-mile  test  course. 

4.4  Twenty-nine  licensed  U.S.  Array  truck 
drivers  were  trained  to  drive  all  three 
types  of  vehicles. 

5.3  The  data  analyses  revealed  that  the  drivers 
Judged  the  3-1/2  ton  and  5-ton  vehicles  to 
be  significantly  better  than  the  2-1/2-ton 
vehicle  from  a  human  factor's  standpoint. 
There  were  no  significant  differences 
between  the  3-1 /2-ton  and  5-ton  trucks. 

5.4  The  evaluation  methodology  should  be  of  use 
to  all  agencies  conducting  user  tests  of 
trucks  and  similar  vehicles. 
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(1)  Command  Climate:  Communication  Flow, 
Decision-Making,  Motivation,  Human 
Resource  Emphasis,  Lower  Level 
Influence. 
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3.3 
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3.5 

Measurement  Procedures 

Supervisory  Leadership:  Support, 
Teamwork,  Goal  Emphasis,  Work 
Facilitation. 

Peer  Leadership:  Support,  Teamwork, 
Work  Facilitation,  Problem  Solving. 

Work  Group  Processes:  Work  Group 
Coordination,  Work  Group  Readiness, 
Work  Group  Discipline. 

Satisfaction 

Integration  of  Men  and  Mission 
Training 
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(8)  General 

(9)  Equal  Opportunity 

(10)  Drug  Abuse 

(11)  Alcoholism  Prevention 

(12)  Community  Interrelationships 
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2.3  The  purpose  of  this  study  was  to  develop  a 
methodology  for  evaluating  the  impact  of 
the  Human  Resource  Management  Support 
System  (HRMSS)  in  those  commands  where  it 
is  utilized.  Its  primary  mission  was  to 
assist  those  commands  with  trained 
specialists  in  order  to  provide  assistance 
in  increasing  the  overall  performance  of 
its  personnel  toward  goal  attainment  and 
quality  performance. 

2.M  Several  limitations  characterized  this 
study: 


f.'i 

“/< 


«■> 
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(1)  Data  samples  used  to  test  the  two 
hypotheses  were  necessarily  smaller 
than  those  required  for  rigorous 
statistical  analysis. 

(2)  Data  used  to  test  both  hypotheses  were 
from  different  units  (test  sites). 

3.1  This  study  centered  around  two  hypotheses: 

(1)  There  will  be  no  significant 
difference  in  response  between  the 
Human  Resources  Management  (HRM) 
specirlists  and  the  commanding 
officers  regarding  how  well  the 
various  Human  Resources  Availability 
(HRAV)  activities  were  accomplished. 

(2)  Comparison  of  units  participating  in 
an  HRAV  with  control  units  will 
statistically  differ  in  a  significant 
manner. 


3.2  Two  questionnaires  were  designed  to  test 
the  first  hypothesis.  One  tapped  HRM 
specialists  regarding  their  assessment  of 
their  own  performance;  also,  the  commanding 
officer  of  the  unit  receiving  the  services 
was  asked  the  same  set  of  questions.  AB5 1 3 
The  second  hypothesis  would  lead  one  to 
expect  some  degree  of  positive  change  in 
unit  performance  as  measured  by  recognized 
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performance  criteria.  This  was  measured  by 
comparison  of  unit  performance  in  the  area 
of  interest  with  the  HRAV  being  the 
independent  variable. 

4.1  A  Likert-type  scale  was  administered  to 

relevant  respondents  and  data  were  analyzed 
through  the  use  of  chi-square  tests  of 
significance  and  regression  equations. 

5.3  It  wa3  concluded  that  the  various  HRAV 
activities  had  merit  to  the  commanding 
officers  and  that  the  HRM  Support  Team 
perform  those  activities  at  an  acceptable 
level  of  effectiveness. 
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This  report  describes  a  systematic, 
statistically-directed  search  for  automated 
flight  measurements  that  correlate  with 
pilot  proficiency. 

In  experiment  1,  four  different  flight 
tasks  of  about  10  minutes  long  were 
measured.  In  experiment  2,  six  additional 
tasks  were  measured. 

Pilots’  performance  during  various 
simulated  flight  tasks  was  required  for 
this  evaluation. 

Three  hundred  and  twenty  six  measurements 
over  the  four  task  series  in  the  first 
experiment  and  2436  measurements  in  the  2nd 
experiment  were  made.  Many  measurements 
had  the  same  name  but  were  measured  in 
different  tasks.  A  numbering  system  was 
used  to  identify  each  of  them. 

Tables  of  more  than  400  impoi  tant 
measurements  were  developed  with  group 
means  and  standard  deviations  and  further 
cross-tabulations  showing  which  tasks  and 
families  of  measurements  best  discriminate 
among  pilots.  Three  different  statistical 
methods  were  used  to  select  a  set  of 
measurements  from  experiment  1  and  combine 
them  into  two  canonical  variables,  each  a 
linear  weighted  combination  of  the 
measurements  in  the  set,  to  discriminate 
optimally  among  the  three  groups  of 
subjects.  Applying  the  canonical  variables 
to  the  repeated  measurements  of  Experiment 
2  allowed  several  deductions  aboi’t  the  best 
selection  procedures  to  be  made. 
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4.3  This  approach  utilized  a  GAT-1"  trainer ,  a 
Line-8  computer  system  and  pilots. 

4.4  Subjects  (3C  in  each  experiment)  were 
selected  on  the  basis  of  their  flying 
experience.  One  group  had  less  than  10 
hours  experience,  the  second  group  had 
25-50  hours,  and  the  third  group  had  more 
than  100  hours  of  experience.  The  majority 
had  had  some  experience  in  the  GAT-1 
trainer.  Some  subjects  volunteered,  others 
were  paid. 

4.5  Before  each  subject  was  asked  to  perform 
the  series  of  maneuvers,  a  warm-up  period 
was  allowed  to  familiarize  pilots  with  the 
GAT-1  and  its  flight  characteristics.  The 
warm-up  period  varied  depending  on  the 
skills  of  the  pilots.  In  experiment  1  the 
major  tasks  were: 

(1)  Roll  and  pitch  tracking. 

(2)  Roll  and  pitch  tracking  with  power 
changes . 

(3)  Flight  profile. 

(4)  ILS  landing  approach. 

In  Experiment  2,  the  following  tasks  were 
added : 

(1)  Roll  tracking. 

(2)  Roll,  pitch  and  yaw  tracking. 

(3)  Reduced  bandwidth  roll  tracking. 

(4)  Reduced  competence  roll  tracking. 

(5)  Ground  reference  turning  maneuver. 

(6)  Altitude  position  tracking. 


5.4  The  results  show  that  there  is  little 

difficulty  in  obtaining  measurements  that 
correlate  significantly  with  experience. 
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2.1  The  purpose  of  this  document  was  to  provide 

and  the  Navy  with  a  generalizable  Test  and 

2.2  Evaluation  (TAE)  planning  scheme  which  can 
be  more  easily  reviewed  and  evaluated  in 
addition  to  providing  a  basis  for 
implementation  monitoring  and  systematic 
data  analysis.  In  brief,  the  generalized 
TAE  plan  describes  the  details  that  the 
Human  Factors  Engineer  (HFE)  should 
consider  when  preparing  or  implementing  a 
plan  for  a  specific  weapon  system.  The 
essential  elements  of  this  system  covered 
planning,  conducting,  reporting,  and 
correcting;  the  system  becomes  more 
detailed  as  the  tasks  and  sub-tasks  become 
smaller  in  scope. 

3.1  The  characteristics  of  the  system  consisted 
of  the  following:  review  of  test  plan 
requirements;  review  general  and/or 
detailed  specifications;  review  previous 
test  data;  prepare  general  test  plan; 
detail  the  test  plan;  negotiate  and/or 
revise  test  plan;  commence  testing  within 
given  constraints;  document  human  factors 
deficiencies;  prepare  the  human  factors 
input  to  project  office  TAE  final  report; 
prepare  separate  report  for  the  Feedback 
Loop  Action  Generation  (FLAG)  information 
system;  negotiate  for  follow-on  human 
factors  effort;  participate  in  product 
improvement  and  deficiency  correcting 
programs. 


Further  Research  Areas 

6.1  Measurement  System 
Limitations 

6.2  Research  Potentials/ 
Priorities 

6.3  Research  Planning 
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3.2  Examples  of  how  the  above  system  components 
were  quantified  are  given  in  detail. 
Examples  of  these  sub-routines,  in 
respective  order,  are  as  follows: 

(1)  Determination  of  phase  and  scope  of 
T&E,  determination  of  scheduling. 

(2)  Budget  constraints,  and  manpower 
requirements. 

(3)  Review  of  weapon  system  documents  for 
relevance  to  human  factors. 

(4)  Review  analytical  efforts,  simula¬ 
tions,  modeling,  etc.,  conducted 
during  concept  validation. 

(5)  List  areas  to  be  investigated. 

(6)  Determine  safety  factors  to  be 
incorporated  into  the  aircraft/weapon 
system. 

Possible  areas  to  be  investigated  may 
consist  of  such  factors  as  crew  station 
geometry,  anthropometries,  and  controls/ 
displays;  moreover,  questionaire  checklist 
tests  must  be  specified  and  instrumentation 
requirements  listed.  Project  office 
personnel  should  be  consulted  regarding 
test  parameters  and  data  collection 
methods;  finally,  the  report  itself  follows 
a  standard  format  which  meets  the 
requirements  of  the  FLAG  system. 
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1.  State  of  the  Art  Review 
of  the  Process 

1.1  Ceneral  System 
Measurements 

1.2  System  Taxonomy 
Model  (STM) 

1.3  Overall  Conceptual 
Process  Model  (CPM) 


1.1  The  basic  theme  of  this  paper  is  that  all 
the  evaluation  models  are  based  on 
variations  in  assumptions  of  liberal 
ideology,  the  conceptions  of  liberal 
democracy.  The  models  differ  from  one 
another  as  the  basic  assumptions  vary. 


2.  Contextual  Components 
of  the  Process 

2.1  System  Definition 

2.2  Mission  Definition 

2.3  Environment  Definition 
2. A  Ceneral  Constraints 

2.5  Performance 
Requirements,  Ultimate 

2.6  Performance 
Criteria,  Ultimate 


The  paper  describes  some  evaluation  models 
and  explains  their  premises.  It  is  part  of 
a  larger  work  which  attempts  to  present  a 
comprehensive  analysis  of  evaluation. 


3.  Analytic  Components 
of  the  Process 

3.1  Practical  Measurable 
Attributes 

3.2  Practical  Attribute 
Measures 

3.3  Performance 
Requirements,  Specific 

3.  A  Performance 

Criteria,  Specific 
3.5  Measurement  Procedures 


A.  Planning  Components 
of  the  Process 
A.1  Analytic  Methods 
A. 2  Parameter  Determinations 
A.  3  Apparatus  for  Testing 
A. A  Personnel  for  Testing 
A. 5  Test  Plans 


5.  Application  Components 
of  the  Process 

5.1  Test  Execution 

5.2  Data  Analysis 

5.3  Findings  Interpretation 
5. A  Conclusions  and 

Recommendations 


6.  Further  Research  Areas 

6.1  Measurement  System 
Limitations 

6.2  Research  Potentials/ 
Priorities 

6.3  Research  Planning 
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1.  State  of  the  Art  Review  2.1 

of  the  Process 

1.1  General  System 
Measurements 

1.2  System  Taxonomy 

Model  (STM)  2.2 

1.3  Overall  Conceptual 
Process  Model  (CPM) 

2.  Contextual  Components 

of  the  Process  2.^ 

2.1  System  Definition 

2.2  Mission  Definition 

2- 3  Environment  Definition 

2.  *1  General  Constraints 

2.5  Performance 
Requirements,  Ultimate 

2.6  Performance 
Criteria,  Ultimate 

2.5 

3.  Analytic  Components 
of  the  Process 

3. 1  Practical  Measurable 
Attributes 

3.2  Practical  Attribute 
Measures 

3.3  Performance  3.1 

Requirements,  Specific 

3. H  Performance 

Criteria,  Specific 

3- 5  Measurement  Procedures  3*2 

0.  Planning  Components 
of  the  Process 
*t.1  Analytic  Methods 

H.2  Parameter  Determinations  **.1 

4.3  Apparatus  for  Testing 

4. 4  Personnel  for  Testing 

A.  5  Test  Plans 

5.  Application  Components 
of  the  Process 

5.1  Test  Execution 

5.2  Data  Analysis 

5.3  Findings  Interpretation 

5-4  Conclusions  and 

Recommendations 

6.  Furtner  Research  Areas 

6.1  Measurement  System 
Limitations 

6.2  Research  Potentiels/ 

Priorities 

6.3  Research  Planning 


This  evaluation  addressed  the  Microwave 
Landing  System  (MLS)  including  aircraft  and 
pilots. 

A  number  of  landing  approach  patterns  with 
multiple  aircraft  were  selected  for  this 
study. 

It  was  noted  that  aircraft  using  the  same 
airspace  are  likely  to  have  a  wide  spectrum 
of  interpretative  capabilities.  The 
constraints  for  use  of  this  equipment  are 
weight,  space,  capital  and  maintenance 
costs . 

The  ultimate  requirement  is  that  the  system 
provide  accurate  information  regarding  an 
aircraft's  position  during  a  landing 
approach. 

Deviation  in  position  and  speed  from 
planned  glide  path  were  measured. 

Errors  across  track,  along  track,  above  and 
below  glide  path  and  speed  along  path  were 
determined. 

The  descent  rates  were  compared  with  the 
criteria  and  the  difference  was  defined  as 
error.  Error  values  were  modified  to 
relate  to  their  size  and  significance.  The 
modified  error  value  was  weighted  for  each 
parameter  to  reflect  its  true  influence 
relative  to  other  parameters.  The  modified 
and  weighted  error  values  were  combined 
into  score(s). 
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A  limited  number  of  typical  approaches  and 
rates  of  descent  were  selected  for  this 
analysis. 

This  scoring  system  works  and  is  being  used 
for  other  MLS  systems. 
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1 .  State  of  the  Art  Review 
of  the  Process 

1.1  General  System 
Measurements 

1.2  System  Taxonomy 
Model  (STM) 

1.3  Overall  Conceptual 
Process  Model  (CPM) 


1.3  The  Phase  III  Computerized  Training  System 
(CTS)  Final  Acceptance  Test  Plan  was 
conducted  in  five  levels  that  corresponded 
to  five  test  objectives: 

Level  I:  Verification  of  Phase  II  Software 
Tests. 


2.  Contextual  Components 
of  the  Process 

2.1  System  Definition 

2.2  Mission  Definition 

2.3  Environment  Definition 
2. A  General  Constraints 

2.5  Performance 
Requirements,  Ultimate 

2.6  Performance 
Criteria,  Ultimate 

3.  Analytic  Components 
of  the  Process 

3.1  Practical  Measurable  2.1 

Attributes 

3-2  Practical  Attribute  _  _ 

Measures 

3.3  Performance 
Requirements,  Specific 

3-A  Performance 

Criteria,  Specific 
3-5  Measurement  Procedures 

A.  Planning  Component; 
of  the  Process 
A.1  Analytic  Methods 
A. 2  Parameter  Determinations 
A. 3  Apparatus  for  Testing 
A. A  Personnel  for  Testing 
A. 5  Test  Plans 

5.  Application  Components 
of  the  Process 

5.1  Test  Execution 

5.2  Data  Analysis 

5.3  Findings  Interpretation 
5. A  Conclusions  and 

Recommendations 


Level  II:  Reliability. 

Level  III:  Verification  of  Phase  II  Class 
I  Language  Tests. 

Level  IV:  Response  Time. 

Level  V:  Full  Operational  Load. 

The  CTS,  a  full  128-terminal  system,  was 
under  investigation  in  the  present  study. 
The  primary  objectives  of  the  test  were  to 
determine  whether  the  full  128-terminal 
system  will  operate  under  operational  live 
user  conditions  and  meet  contract 
requirements  for  reliability  and  response 
time. 

As  regards  reliability  the  system  must 
achieve  a  minimum  of  95%  under  specified 
conditions.  Up-tirae  refers  to  the 
capability  of  the  hardware  and  software  to 
perform  basic  tasks  of  system  initiation, 
program  compilation  and  execution,  job  and 
file  management,  and  program  library 
maintenance. 


6.  Further  Research  Areas 

6.1  Measurement  System 
Limitations 

6.2  Research  Potentials/ 
Priorities 

6.3  Research  Planning 
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Regarding  response  time,  performance 
specifications  for  the  128-system  state 
that  there  shall  be  a  90%  probability  that 
the  response  time  will  be  2  seconds  or 
less. 

5.4  Since  the  system  did  not  meet  reliability 
objectives  (plus  had  an  insufficient  number 
of  response  time  recordings  for  a 
representative  sample) ,  it  is  recommended 
that  the  system  not  be  accepted  in  its 
present  state.  Following  debugging  of  the 
present  problems,  further  retesting  of  the 
system  for  overall  performance  is 
recommended. 
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State  of  the  Art  Review  1.3 

of  the  Process 

1 . 1  General  System 
Measurements 

1.2  System  Taxonomy 
Model  (STM) 

1.3  Overall  Conceptual 
Process  Model  (CPM) 

Contextual  Components 
of  the  Process 

2.1  System  Definition 

2.2  Mission  Definition 

2.3  Environment  Definition 

2 . 4  General  Constraints 

2.5  Performance 
Requirements,  Ultimate 

2.6  Performance 

Criteria,  Ultimate  2.1 

Analytic  Components 
of  the  Process 

3.1  Practical  Measurable 
Attributes 

3.2  Practical  Attribute 
Measures 

3.3  Performance 

Requirements,  Specific  2.2 

3. A  Performance 

Criteria,  Specific 
3.5  Measurement  Procedures 

Planning  Components 

of  the  Process 

A.1  Analytic  Methods 

A.  2  Parameter  Determinations  3*1 

A.  3  Apparatus  for  Testing 

A. A  Personnel  for  Testing 

A. 5  Test  Plans 

Application  Components 
of  the  Process 

5.1  Test  Execution 

5.2  Data  Analysis 

5.3  Findings  Interpretation 
5. A  Conclusions  and 

Recommendations 


The  Computer  Aided  Function  Allocation  and 
Evaluation  System  (CAFES)  is  a  support  tool 
for  Human  Factor  Engineers  (HFE)  conducting 
man-machine  research,  requirements 
analysis,  design,  test,  and  training  and 
maintenance  systems  development.  The  CAFES 
program's  principle  objective  is  to 
facilitate  application  of  essential 
elements  of  human  factors  technology  in 
systems  development  using  automatic  data 
processing  techniques  in  order  to  analyze 
and  evaluate  crew  subsystem  performance  as 
it  affects  total  systems  effectiveness. 

The  CAFES  was  being  developed  to  provide 
the  HFE  with  an  integrated  system  of 
computer  models  which  progress  from  early 
concept  formulation  phase  through  crew 
station  design  to  the  final  test  and 
evaluation  of  the  completed  system. 

The  CAFES  objective  was  to  allow  the  HFE  to 
treat  in  a  comprehensive  way  all  parameters 
to  be  considered  in  the  designing  of  a 
man-machine  interface  of  advanced  Navy 
systems. 

The  CAFES  models  were,  in  fact,  composed  of 
six  integrated  sub-models: 

(1)  A  Data  Management  System  (DMS)  which 
provides  baseline  data  for  all  the 
models. 
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(2)  A  Function  Allocation  Model  (FAM), 
which  is  a  collection  of  computerized 
algorithms  that  helps  derive  and 
process  various  alternatives  for 
allocating  function?  to  operator (o) 
and  machines. 

(3)  A  Workload  Assessment  Model  (WAM) 
which  considers  the  human  performance 
aspect  of  man-machine  function 
allocation  schemes  on  a  time  and 
cumulative  tasks  basis  to  determine  if 
man  can  perform  all  of  the  tasks 
derived  from  the  functions  that 
compose  one  or  more  missions.  The 
model  also  determines  those  periods 
when  man  is  overloaded  in  terms  of 
time  availability  versus  time  required 
to  do  all  tasks. 

(4)  A  Computer-Aided  Design  (CAD)  which 
will  furnish  aids  to  the  designer  in 
producing  crew  station  configurations 
that  are  consistent  with  mission 
requirements. 

(5)  A  Crew  Station  Geometry  Evaluation 
(CGE)  which  evaluates  the  physical 
geometry  of  a  crew  station. 

(6)  A  Human  Operator  Simulator  (HOS)  which 
provides  a  generalized  model  of  a 
seated  human  being  in  a  task 
environment  with  limitations  on  time 
and  goals. 

3.3  The  specific  performance  requirements  of 

the  above  models  were  as  follows: 

(1)  DMS:  its  major  functions  are  data 
input  and  storage,  file  modification, 
CAFES  Executive,  error  diagnostics, 
report  generation. 

(2)  FAM:  predicts  overall  system 
effectiveness  and  generates  crew 
oprrational  procedures  for  detailed 
evaluation  of  promising  allocation 
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candidates. 

(3)  WAM:  crew  workload  is  analyzed  on  the 
basis  of  how  workload  varies  in  each 
performance  "channel"  (vision,  hands, 
feet,  etc.). 

(*1)  CAD:  assists  in  crew  station  geometry 
development,  design  analysis,  and 
design  drawings. 

(5)  CGE:  functions  includes  man-model 
simulation  (computes  body 
configurations  approximating  human 
movements  during  task  taxonomy), 
conducts  interference  analysis 
(identifies  visual  or  physical 
interference  for  various  crew  members 
executing  a  task  sequence),  evaluates 
military  standard  and  specification 
compliance. 

(6)  HOS:  serves  to  simulate  human 
operator  performance  in  aircraft 
systems,  interfaces  directly  with  the 
hardware  simulator,  provides  simulated 
human  performance  output,  and 
re-examines  human  performance  workload 
and  task  duration  in  an  operating 
environment. 

5.*J  Computer  techniques  can  only  assist  the 

HFE,  not  substitute  for  his  judgment.  The 
computer  can  assist  the  engineer  in  design 
and  analysis  tasks  by  recalling/processing 
data  and  performing  simple  logic 
operations.  The  responsibility  of  the  HFE 
remains  with  interpretation  of  effects  from 
system  requirements,  performance  criteria, 
and  trade  study  results  in  defining  system 
specifications. 
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The  system  being  studied  was  the  Advanced 
Simulator  for  Pilot  Training  (ASPT) 
configured  as  the  T-37  aircraft. 

The  mission  included  five  specific  flight 
maneuvers:  takeoff,  GCA,  360  degree 
overhead  patterns,  aileron  rolls,  and  slow 
flights. 

The  environmental  conditions  in  this  study 
included  three  levels  of  wind,  three  levels 
of  turbulence,  and  two  levels  of 
ceiling/visibility. 

The  constraints  on  the  system  were  those  of 
the  ASPT  which  include  six  degrees  of 
freedom  motion  platforms  providing 
approximately  three  feet  of  vertical  travel 
and  four  feet  of  horizontal  travel. 
Displacement  capabilities  included:  pitch 
-  20  degrees  to  +  30  degrees;  roll  ♦  22 
degrees;  yaw  +  32  degrees.  The  pneumatic 
G-seat  provided  more  continuous  cues  than 
the  motion  platform  in  response  to 
requirements  for  each  particular  maneuver. 
The  visual  system  was  comprised  of  seven 
36-inch  monochromatic  CRT's  giving  the 
pilot  110  degrees  to  -40  degrees  vert’ cal 
cuing  and  +  150  degrees  horizontal. 

The  dependent  variables  used  in  this  study 
were  derived  from  the  ASPT  Automated 
Performance  Measurement  System  which  is  a 
criterion-referenced  approach  to 
measurement.  Because  most  skillful 
piloting  involves  the  attempt  to  maintain 
or  change  to  specified  flight  parameters 
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criteria,  deviations  from  these  desirr J 
parameters  provides  the  method  of 
quantitative  objective  performance 
measurement.  Some  examples  of  the 
dependent  variables  (measures)  reported  in 
this  study  include: 

(1)  Deviations  from  prescribed  flight 
parameters. 

(2)  Power  applied  to  control  surfaces. 

(3)  Scores  formulated  from  multiple 
factors. 

3.5  Each  subject  flew  one  profile  72  times  and 
the  other  27  times  during  the  course  of  the 
study.  The  profiles  were  randomly  ordered 
for  all  subjects.  Each  session  was  begun 
with  instructions  provided  by  a  computer 
driven  word  generator.  Each  maneuver  was 
begun  on  command  and  completed  when 
selected  criteria  were  satisfied.  At  the 
completion  of  each  maneuver  within  the 
profile,  the  console  operator  entered 
comments  on  any  system  malfunction  or 
errors  experienced  during  the  maneuver. 

All  other  data  values  were  recorded  by  the 
ASPT  computer  at  an  iteration  rate  of  3.75 
to  15  times  per  second. 

4.1  A  multivariate  analysis  of  variance  was 
used  for  data  analyses.  The  statistic  used 
in  determining  significant  effects  was  the 
Wilks  Lambda. 

4.2  The  independent  variables  of  the  test  were 
as  follows: 

(1)  Wind  (0,  12,  24  knots  from  60°) 

(2)  Turbulence  (none,  light,  moderate) 

(3)  Ceiling/visibility  (clear  and  minimum 
(200  ft/1/2  mile)). 
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4.3  The  primary  test  apparatus  was  the 
six-degree-of-freedom  Advanced  Simulator 
for  Pilot  Training  configured  as  a  T-37 
aircraft  cockpit. 

4.4  Tl.e  test  subjects  were  three  T-37 
instructor  pilots  whose  flying  time  ranged 
from  550  to  900  hours. 

4.5  Two  separate  experimental  designs  were 
used.  The  first  was  structured  to  evaluate 
main,  first-order  interaction,  and 
second-order  interaction  effects  of  all  six 
independent  variables.  The  second  design 
was  a  partially  compounded  factorial  which 
used  four  independent  variables,  each  with 
three  levels. 

5.1  Subjects  were  scheduled  on  a  day-to-day 
basis  depending  on  ASPT  system 
availability. 

5. 4  In  this  study,  each  of  the  system  configura¬ 

tion  variables  produced  significant  ef¬ 
fects.  The  dependent  variables  used  to 
measure  performance  in  this  study  showed 
that  manipulation  of  the  three  environmen¬ 
tal  variable  combinations  produced  changes 
in  system-oriented  dependent  variables. 
Similarly,  changes  in  the  pilot  input 
variable  were  concomitant  with  simulator 
configuration  changes. 
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1.1  This  report  represents  an  initial  attempt 
to  scope,  through  a  literature  review,  the 
complexity  of  developing  a  conceptual 
structure  (model)  of  operator  workload. 

The  ultimate  goal  is  to  develop  a 
quantitative  index  of  operator  performance 
for  any  point  in  time  during  a  given 
vehicle  operation.  The  operator's  role  is 
basically  that  of  a  data  transmission  and 
processing  link  between  displays  and 
controls  of  any  vehicle.  The  operator's 
basic  functions  are  sensing,  identifying, 
and  interpreting. 

Workload  is  the  extent  to  which  an  operator 
is  occupied  by  a  task.  It  can  be  divided 
into  three  functionally  relatable 
attributes:  input  load,  operator  effort, 
and  work  result. 

Four  categories  of  workload  research  were 
identified  during  the  literature  review: 

(1)  Time-and-motion  studies. 


4.  Planning  Components  (2) 

of  the  Process 
1) .  1  Analytic  Methods 

4.2  Parameter  Determinations  (3) 

4.3  Apparatus  for  Testing 

4.4  Personnel  for  Testing 

4.5  Test  Plans 


Information  processing  status. 
Operator  activation-level  studies. 
Equipment  design  -  implicit  studies. 


5-  Application  Components 
of  the  Process 

5.1  Test  Execution 

5.2  Data  Analysis 

5.3  Findings  Interpretation 

5.4  Conclusions  and 
Recommendations 


The  report  describes  each  in  detail  and 
presents  their  advantages  and  disadvan¬ 
tages. 


6.  Further  Research  Areas 

6.1  Measurement  System 
Limitations 

6.2  Research  Potentials/ 
Priorities 

6.3  Research  Planning 
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The  report  concludes  from  the  literature 
survey  that,  in  general,  the  approach  to 
workload  research  has  been  either  too 
molecular  or  too  molar  to  provide  the  broad 
spectrum  of  data  required  during  various 
phases  of  crew  system  design  and  evalu¬ 
ation.  A  number  of  potentially  useful 
techniques  are  available  to  partially 
provide  meaningful,  quantitative  answers  on 
many  parameters  influencing  operator  ef¬ 
fort.  These  techniques  need  to  be  systemat¬ 
ically  evaluated  and  integrated  in  the 
specific  context  of  design  requirements. 
Because  of  the  complexity  of  factors 
affecting  human  behavior  and  our  limited 
knowledge  regarding  operator  capabilities 
and  limitations,  no  absolute  measure  of 
workload  exists. 
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This  study  describes  a  comprehensive  crew 
system  development  tool  called  CAFES 
(Computer  Aided  Function  Allocation  and 
Evaluation  System).  The  CAFES  concept  is  a 
set  of  submodels  working  in  conjunction 
with  a  data/information  management  system. 
Each  submodel  may  be  used  individually  or 
in  combination  with  one  or  more  of  the 
others  depending  upon  the  data  requirements 
(and  system  definition).  The  modular 
nature  of  CAFES  provides  flexibility  for 
integrating  the  parameter  in  the  definition 
and  evaluation  of  the  human  role  in  naval 
aircraft  systems.  The  mission  of  CAFES  is 
a  reduction  in  performance  time  and 
increased  quantification  for  the  following 
human  factor  tasks: 

(1)  Perform  and  evaluate  man-machine 
function  allocations. 

(2)  Develop  design  concepts/requirements 
for  each  crew  work  station. 

(3)  Identify  potential  human  factors 
engineering  risks. 

The  following  are  descriptions  of  the 
various  submodels  which  make  up  CAFES: 

(1)  The  Functional  Allocation  Model  (FAM) 
is  a  collection  of  computerized 
algorithms  that  will,  in  conjunction 
with  the  Data  Management  System  (DMS) 
and  crew  system  designer,  derive  and 
process  various  alternatives  for 
allocating  functions  to  operator (s) 
and  machines.  The  FAM  will  identify 
and  organize  system  function  to  an 
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allocable  level,  identify  and  rank 
order  function  allocation  schemes,  and 
will  provide  detailed  communication 
with  the  available  system  definition 
at  any  point  in  the  development  cycle. 

(2)  The  Workload  Assessement  Model  (WAM) 
considers  the  human  performance 
aspects  of  man-machine  function 
allocation  schemes  to  determine 
whether  man  can  perform  all  of  the 
tasks  derived  from  the  allocable 
functions  that  make  up  one  or  more 
missions.  The  submodel  uses  a 
timeline  of  mission  duration  and 
determines  those  periods  when  man  is 
overloaded,  necessitating  either  a 
reallocation  of  that  function  to 
machine  or  additional  crew  or  a 
modification  of  the  system 
requirement. 

(3)  The  Computer-Aided  Design  (CAD) 
furnishes  aids  with  which  the  designer 
can  develop  crewstation  configurations 
compatible  with  the  men  who  will  work 
in  them,  consistent  with  the  missions 
to  be  performed,  and  constrained  by 
military  design  standards  and 
specifications  as  well  as  technical 
and  cost  considerations. 

(4)  The  Crewstation  Geometry  Evaluation 
(CGE)  program  is  a  standardized  method 
for  evaluating  the  physical  geometry 
of  a  crewstation.  It  evaluates  the 
physical  compatibility  of  a  seated 
crewmember  of  any  size  with  any 
crewstation  as  soon  as  a  design 
concept  is  available. 

(5)  The  Human  Operator  Simulator  (HOS)  is 
a  generalized  model  of  a  seated  human 
being  in  a  time-based,  goal-oriented, 
task-processing  environment.  It  is 
used  in  parallel  with  the  CGE  submodel 
in  the  CAFES  system  to  evaluate  the 
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function  allocation  as  well  as  the 
adequacy  of  the  crewstetion  design. 

It  determines  operator  workload  as 
well  as  procedure  (task)  duration, 
movement  distance,  and  sequence 
statistics  going  from  one  control  to 
the  next. 

(6)  The  Tradeoff  Analysis  Routine  (TAR)  is 
used  after  the  CGE  and  HOS  models  have 
been  run  and  modification  to  the 
weapon  system  are  necessary  from  a 
geometric  or  workload  point  of  view. 
The  crew  systems  designer  must  decide 
whether  it  would  be  most  effective  to 
modify  the  design  to  eliminate  the 
problem  or  to  reallocate  one  or  more 
functions  to  eliminate  the  problem. 
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1.1  Methods  pertaining  to  the  measurement  of 
criteria  and  ascertaining  underlying 
criterion  constructs  were  reviewed.  Three 
criterion  measurement  models,  i.e.,  the 
ultimate  criterion  model,  the  multiple 
criterion  model,  and  a  "general"  criterion 
model  for  the  determinants  of  managerial 
effectiveness,  were  examined  and  attempts 
made  toward  integration.  The  three  models 
were  compared  to  a  formal  construct 
validation  model,  and  strengths  and 
weaknesses  in  both  the  constructs  provided 
by  each  of  the  criterion  models  and 
construct  validation  procedures  were 
discussed.  An  integrated  multiple  and 
general  criterion  model  and  construct 
validation  procedures  more  extensive  than 
the  multitrait-multimethod  matrix  were 
recommended  for  future  criterion  research. 
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1.1  This  report  describes  a  plan  for  testing 
the  Mechanized  Infantry  Combat  Vehicle 
(MICV) . 

2.1  The  MICV  system  is  a  companion  to  the  main 
battle  tank  and  is  composed  of  four  parts: 
the  vehicle  itself,  the  primary  armament, 
the  secondary  armament,  and  the 
supplementary  armament. 

2.2  The  vehicle's  ultimate  functions  are  as 
follows:  mounted  fighting  capability  for 
one  mechanized  infantry  squad;  swim  and 
cross-country  mobility;  full  protection  of 
the  infantry  squad  from  small  arms;  etc. 
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The  MICV  system  is  to  be  tested  under 
certain  climate  conditions  mainly 
characterized  by  "winter  thaw." 

The  primary  role  of  the  MICV  lies  in  its 
offensive  weapons  capability. 

In  this  role,  the  MICV  is  expected  to 
improve  mobility,  protection,  firepower, 
survivability,  reliability,  crew  comfort, 
etc. 

Characteristics  of  the  system  which  will  be 
measured  include:  mission  performance, 
survivability,  availability,  training,  and 
logistics. 

Measurement  of  the  above  may  be  performed 
by  measuring  the  speed,  range,  and 
maneuverability  degradations  caused  by 
snow,  mud,  and  frozen  ruts  (for  "mission 
performance");  ability  of  MICV  to  be 
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effectively  camouflaged  (for  surviv¬ 
ability);  ability  of  crew  to  perform 
maintenance  and  service  of  the  MICV  (for 
availability);  techniques  of  vehicle 
operation  (for  training),  etc. 

Test  parameters  include  day  and  night 
conditions  and  live  fire  conditions. 
Organization,  doctrine,  training,  and 
limited  logistical  support  will  be  factors 
held  constant;  terrain  and  weather  will  be 
uncontrolled. 

Data  collected  will  be  analyzed  and 
reported  in  relative  terms  (not  quantita¬ 
tive).  RAH  data  will  be  qualitative; 
reliability  will  be  limited  to  a  consolida¬ 
tion  of  data  regarding  malfunctions  which 
occur,  impact  of  malfunctions,  and  mainte¬ 
nance  problems  associated  with  them. 
Availability  will  be  measured  through 
records  on  uptime  and  downtime  of  system. 
Maintainability  will  be  observed  under 
winter  thaw  conditions  and  will  be 
established  through  observations  of  the 
test  experience. 

Participants  will  perform  mechanized 
infantry  squad  and  scout  team  tactics  as 
prescribed. 


Johnson,  E.M.  A  Baker,  J.D.  Field  testing:  The  delicate  compromise. 
Human  Factors,  1974,  16(3),  203-214. 
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This  paper  selectively  compares  human 
factors  field  testing  with  laboratory  ex¬ 
perimentation.  The  basic  steps  of  research 
design  provide  a  point  of  departure  to 
illustrate  the  differences  between  labora¬ 
tory  and  field  research.  Differences  in 
problem  recognition  are  described.  It  is 
noted  that  the  laboratory  researcher  often 
uncovers  a  research  problem  by  exhaustive 
examination  of  the  literature.  In  contrast, 
a  problem  for  the  field  researcher  typically 
originates  with  questions  from  a  sponsor/ 
user  group.  A  table  is  presented  which 
charts  the  stages  of  a  system’s  life  and 
ties  these  stages  to  the  associated  events 
that  aid  problem  recognition. 

It  is  evident  that  the  precision  of  the 
evaluation  data  is  a  direct  functior  of  the 
stage  of  life  at  which  the  problem  is 
identified . 

Problem  recognition  is  only  the  first  link 
in  the  chain  of  thought  necessary  for 
planning  a  successful  research  study.  The 
next  step  of  the  experimental  method  should 
follow  in  a  standard  order. 

The  differences  between  laboratory  and  field 
testing  in  the  basic  steps  of  a  research 
design  are  presented  on  the  following  page. 

While  it  can  be  noted  that  there  are  great 
similarities  between  laboratory  experimenta¬ 
tion  and  field  research,  many  subtle,  but 
critical,  differences  exist  in  the  substance. 
Field  testing  is  not  a  simple  extension  of 
the  laboratory  into  an  operational  setting. 
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Major  Differences  in  Hypotheses 


Characteristic 


Laboratory 


Type  of  Study 


Derivation 

Definition 

Nunber 

Type  of  Test 
Purpose 


literature  search 

Theory/prior  data 

Deductive 

Precise 

Few 

Null  hypothesis 

levelop  further  research/theory 


Colleagues  and  institutional  reports 

System  model 

Inductive 

Vague 

Many 

Positive  hypothesis 
Answer  a  question 


Characteristic 


Selection 

Number 

Relationship  Between 

Definition 

Durability 


Laboratory 


Ad  hoc;  experimenter  determined 
Few 

Orthogona  1 

Precise 

High 


Type  of  Study 


Post  hoc;  system  determined 
Many 

Nonorthogonal 

Vague  (easily  confused  with  dependent  varlat  es 
Low-obsolescent 


Major  Differences  in  Dependent  Variables 


Characteristic 


Measures 

Definition 

Nu-ter 

Relationship  Between 
Type  of  Measure 
Sensitivity 
Desirable  Sensitivity 


Laboratory 

Ad  hoe;  experimenter  determined 

Human  performance 

Precise 

Few 

Independent 

Quantitative 

Linked  to  independent  variable 
Statistical 


Type  of  Study 


Post  hoc;  syste.r  determined 

Human  performance/syster.  performance 

Vague  (easily  confused  with  independent  variables) 

Many 

Unknown  intercorrelation 

Often  qualitative 

Linked  to  system  performance 

Practical 


Major  Differences  in  Control 


Type  of  Study 


Characteristic 

Experimental  Error 
Experimental  Controls 
Protocols 
Intrusive  Factors 
Physical  Environment 
Time 

Scale  (Logistics) 
Va-iableu  Driver,  By 


Laboratory  Field 

Replications  "One  trial" 

Matched  groups/control  groups  Usually  one  group 

Hell  defined  Vague 

"Controlled"  Uncontrolled 

Controlled  Natural 

Broken  into  short  segments  (trials)  Continuous-mission/scenario  shift  time 
Small  Large  , 

Experimenter  System 


Characteristic 


Source 

Motivation 

Attitude 


Laboratory 


"College  Sophomore" 

Implicit 

Positive 


Type  of  Study 


User  Population 
Explicit 

Neutral  (or  negative) 


Characteristic 


Frequency  of  Report 
Audience 

Dissemination 

Ethics 

Result 


Laboratory 


Low— determined  by  quality 
One — professional  community 

Wide— professional  Journal 

Individual 

"Publish  or  perish" 


Type  of  Study 


High — determined  by  cost 

Many:  user /sponsor ;  management;  professional 
community 

Narrow — institutional  report 

Organizational 

•Publish  and  perish" 
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Techniques,  procedures  and  research 
strategies  differ.  However,  a  technology 
is  evolving  to  bridge  the  gap  between  the 
two  research  domains.  Recognizing  the 
problem  and  describing  its  dimensions  does 
not  solve  it.  There  are  few  reports  of 
attempts  to  validate  laboratory  findings 
in  the  field.  Some  indicate  success,  others 
indicate  failure. 

6.1  Limitations  in  the  methods  and  data  of 

human  factors  tend  to  prevent  the  accurate 
prediction  of  human  performance  in  opera¬ 
tional  settings.  What  still  is  required 
is  an  approach  to  research  design  which 
will,  in  fact,  coordinate  the  laboratory 
and  the  field. 
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This  report  describes  the  development  of 
two  docunents:  "Human  Factors  Evaluation 
Data  for  General  Equipment"  (HEDGE)  and 
"Guidebook  Supplement."  These  products 
were  designed  to  provide  test  engineers 
with  a  human  factors  evaluation  guide  to 
potential  human  factors  problems  for  the 
various  types  of  equipment  tested  and  a 
methodology  for  conducting  these  evalu¬ 
ations.  These  documents  were  designed 
specifically  for  use  by  personnel  with 
engineering  qualifications  who  have  had 
little  or  no  exposure  to  human  factors 
evaluation  techniques. 

The  work  was  conducted  in  three  phases. 
Phase  I  consisted  of  defining  requirements 
for  human  factors  testing  and  analyzing 
current  test  and  evaluation  practices.  A 
classification  system  for  test  items  and 
test  functions  and  an  exhaustive  listing  of 
the  requirements  and  constraints  for  the 
contents  and  format  of  the  human  factors 
guide  were  developed. 

Phase  II  developed  the  format  and  content 
of  the  guide.  The  HEDGE  document  consists 
of  four  major  sections:  (1)  how  to  use 
HEDGE,  (2)  an  index  and  associated 
definitions,  (3)  detailed  data  sheets,  and 
(4)  appendix. 

The  detailed  data  sheets  are  the  heart  of 
the  strategy  proposed.  Since  the  major 
emphasis  in  human  factors  evaluation  is  the 
nature  of  the  interaction  between  man 
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and  the  item,  these  sheets  attempt  to 
detail  the  man/item  tasks  that  must  be 
performed  on  all  (or  most)  items  in  r  given 
subclass  for  a  specific  test  function. 

The  second  part  of  these  data  sheets  is  a 
detailed  design  consideration  sheet.  These 
pages  contain  the  critical  human  factors 
considerations  in  the  test  and  evaluation 
of  the  class  of  items  for  which  they  were 
prepared. 

Phase  III  of  this  project  was  devoted  to 
the  evaluation  and  revision  of  the 
products.  This  evaluation  involved  on-site 
use  and  evaluation  by  test  engineers  and 
the  conduct  of  field  tests. 

Based  on  these  user  trials  it  was  deter¬ 
mined  that  these  documents  are  appropriate 
and  adequate  for  human  engineering  evalu¬ 
ations  of  general  military  equipment. 
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The  Human  Resources  Test  and  Evaluation 
System  (HRTES)  is  a  systematic  and  integra¬ 
tive  approach  to  planning  and  conducting 
evaluations  of  human  contributions  to  sys¬ 
tem  performance.  It  encompasses  a  set  of 
procedures  which  will  assure  that  human 
resources  (1)  are  properly  included  in  a 
system  design,  and  (2)  are  adequately 
assessed  and  evaluated  during  Operational 
Test  and  Evaluation  (OT  &  E). 

The  HRTES  philosophy  is  that  understanding 
of  missions  is  basic  to  the  measurement  of 
systems  in  operational  tests.  There  must 
be  a  logical  link  between  the  missions  to 
be  performed  and  the  selected  measures  of 
performance.  The  first  step  in  the  HRTES 
procedure  to  accomplish  this  linking  is  to 
define  systems  according  to  their  generic 
elass(es).  Each  generic  class  is  defined 
by  general  functional  and  hardware 
similarities.  Systems  belonging  to  the 
same  generic  class  have  certain  missions  in 
common  while  having  other  missions  specific 
to  themselves  individually. 

The  generic  classification  or  indexing  of 
the  system  is  the  first  step  in  the  HRTES 
process.  '  The  subsequent  steps  can  be 
listed  as  follows:  (2)  assignment  of 
missions,  (3)  specification  of  system 
performance  issues,  (4)  identification  of 
human  performance  functions  and  husan 
performance  measures,  (5)  identification  of 
test  conditions,  (6)  specification  of  human 
resources  issues  and  human  resources 
measures,  (7)  operational  testing,  (8) 
evaluation  of  OT  results,  and  (9)  diagnosis 
of  performance  inadequacies. 
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.1  The  prototype  HRTES  handbook  Tists  eleven 
(11)  generic  classes  of  Army  Systems,  VIZ, 
Armored  Vehicles;  Anti-Armor  Weapons;  Tube 
Artillery;  Missile  Artillery;  Aviation;  Air 
Defense  Systems;  Point  Target  Weapons;  Area 
Weapons;  Electronic  Warfare  Systems; 
Electronics  Associated  with  Other  Systems; 
Engineering  Systems. 

.2  Following  generic  classification  of  the 
system,  the  analyst  using  the  HRTES 
procedure  identifies  and  assesses  the 
relative  importance  of  the  various  missions 
of  the  system.  Knowledge  of  the  missions 
allows  the  analyst  to  specify  the  system 
performance  issues  of  interest. 

.4  Specific  test  conditions  (or  constraints) 
that  may  affect  or  further  define  the 
system  performance  issues  must  be 
identified. 

.1  Human  performance  functions  that  derive 
from  the  system  performance  issues  and 
human  resource  issues  that  contribute  to 
human  performance  constitute  the  measurable 
attributes  of  concern  to  the  HRTES 
procedure. 
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A  benchmark  is  a  routine  which  is  run  on  a 
number  of  different  computer  configurations 
to  obtain  comparative  throughput 
performance  figures  regarding  the  abilities 
of  the  various  configurations  to  handle 
some  specific  application.  A  benchmark 
problem  is  a  selected  portion  of  an  entire 
job  that  should  be  representative  of  that 
job. 

The  benchmark  programs  measured  system 
performance  under  the  following 
environmental  conditions: 

(1)  Stand-alone  -  only  one  object  program 
makes  demands  upon  a  system's 
resources . 

(2)  Benchmark  -  the  seven  programs  are 
considered  users,  and  run 
simultaneously  as  a  simulation  of  a 
typical  user  population  to  assess 
system  changes  in  a  real-world 
environment. 

(3)  Real-world  -  one  benchmark  program  is 
run  as  a  pseudo-user  when  the  system 
is  operating  near  its  rated  capacity 
under  conditions  of  the  real-world. 

Planning  and  estimating  the  system's 
workload  was  difficult  due  to  a  variety  of 
demands  and  the  random  user  times. 
Establishing  equivalent  environments  within 
the  computer  systems  being  evaluated  was 
the  major  constraint  of  benchmarks. 
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Because  of  the  factors  stated  -above,  the 
behavior  of  certain  functional  variables  of 
a  system  was  measured;  a  functional 
variable  included  software,  hardware,  and 
user  components  that  support  a  particular 
activity.  The  particular  variables 
measured  were: 

-  swap  activity  -  I/O  activity 

-  compute  activity  -  page  activity 

-  interactive  -  resource 

activity  allocation 

-  user  population 

The  following  were  measures  of  the  stimuli: 

For  compute  activity,  the  number  of  loops 
is  the  measure  of  the  amount  of  CPU  time 
allocated  to  the  program  by  the  scheduling 
algorithm.  The  interactive  activity  used 
the  average  number  of  messages  printed  per 
minute  as  its  measurement.  I/O  activity 
used  the  average  number  of  records  read 
from  the  disc  per  minute  as  its 
measurement.  No  direct  measure  of  swap 
activity  was  made.  Instead,  the  obtained 
results  from  the  small  and  large  versions 
of  the  same  program  were  compared  with  the 
measurement  of  the  increase  in  system 
overhead  due  to  the  increase  in  swap  times. 
The  metric  used  in  evaluating  the  effect  of 
page  management  schemes  was  the  compute 
activity  and  interactive  activity  measures. 
An  increase  in  these  values  implied  a  more 
effective  page  management  scheme.  For 
resource  allocation  activity,  a  measure  of 
the  overall  effect  of  the  resource 
management  algorithms  may  be  determined  by 
running  an  environment  consisting  of  only 
the  benchmark  programs.  The  effect  of 
different  algorithms  can  be  determined  by 
changes  in  the  measures  printed  by  each 
benchmark  program. 

The  measurement  of  compute  activity  was  the 
maximian  throughput  that  any  compute-bound 
job  could  obtain.  The  measurement  of 
response  time  was  the  minimum  (best)  time 
that  any  terminal  bound  Job  could  attain. 


Karush,  A.D.  Two  approaches  for  measuring  the  performance  of  time-sharing 
systems  (SP-3364).  Santa  Monica,  CA:  System  Development  Corporation,  May 
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The  paper  discusses  two  significantly 
different  approaches  for  measuring  the 
performance  of  time-sharing  systems.  The 
"analytical1*  approach  involves  the 
insertion  of  probes  into  the  system  to 
allow  measurement  and  recording  of  the 
system's  most  subtle  behavior.  In  the 
"stimulus"  approach  the  system  is 
conceptualized  as  a  "black  box"  containing 
a  limited  number  of  known  functions.  This 
measurement  technique  involves  applying  a 
controlled  set  of  stimuli  to  the  black  box 
in  order  to  activate  its  function  and  then 
observing  the  results. 

The  analytical  approach  can  have  several 
applications.  In  program  and  hardware 
analysis,  the  stochastic  behavior  exhibited 
by  the  system  and  its  interpretation  in 
terms  of  the  executive  or  supervisor 
function  may  be  elucidated.  In  system 
analysis  the  degree  and  manner  in  which  the 
man-machine  software  complex  has  affected 
its  environment  in  terms  of  costs,  effi¬ 
ciency  and  security  can  be  determined. 

There  are  two  paths  that  can  be  followed  in 
system  research:  (1)  modeling  or  simula¬ 
tion;  and  (2)  controlled  experimentation 
upon  the  time-sharing  itself.  The  stimulus 
approach  can  be  applied  to  several  applica¬ 
tions.  It  can  be  used  to  indicate  the 
degree  of  throughput  and  response  time 
degradation  under  varying  system  loads;  it 
can  indicate  the  degree  to  which  the  system 
load  varies;  it  can  "take  a  reading"  of  the 
system  with  respect  to  the  volume  of 
service  being  requested  from  it;  it  can  bean 
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experimental  test  bed  for  examining  the 
effects  of  different  scheduling  algorithms; 
and  it  can  examine  the  effects  of  changes 
to  functional  variables. 

3.5  The  analytical  approach  has  several 
practical  measurement  options:  (a) 
sampling  measurement  -  this  technique  can 
provide  a  frequency  distribution  which 
describes  the  activity  of  the  program;  (b) 
trace  measurement  -  here  the  interest  is 
often  focused  upon  the  sequential  behavior 
of  some  portion  of  the  system;  (c) 
accounting  measurement  -  this  is  a 
frequently  required  measure  of  a 
time-sharing  system.  It  is  a  summary  of 
resource  utilization  such  as  volume  of  file 
storage,  size  of  programs,  etc.;  (d) 
logical  measurement  -  the  purpose  of  which 
is  to  select  for  further  evaluation  a 
subset  of  a  data  base  as  a  function  of  the 
content,  description,  or  environment  of 
that  data  base;  (e)  playback  measurement  - 
involves  recording  all  the  input  to  the 
system  or  subsystem  together  with  time 
lags,  and  then  rerunning  the  component  of 
interest  with  the  recorded  input  substi¬ 
tuted  for  the  real  input. 

The  stimulus  approach  also  has  several 
practical  measurement  options  depending 
upon  the  environment  in  which  the  system 
runs;  (a)  stand-alone  environment  -  the 
system  is  viewed  as  a  batch  processor  with 
a  benchmark  program  as  the  only  program 
running  which  provides  the  best  measure  of 
throughput  and  response  time;  (b)  benchmark 
environment  -  in  this  situation  all 
benchmark  programs  are  run  simultaneously 
simulating  a  "typical"  user  population;  (c) 
real  world  environment  -  this  technique 
involves  loading  the  system  with  an  almost 
full  complement  of  users  and  running  a 
benchmark  program  simulating  a  user  who  has 
a  constant  and  known  demand  for  service. 
Several  runs  are  needed  to  establish  norms. 
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5.1  Both  the  "analytic"  and  "stimulus" 

approaches  were  used  in  SDC's  ADEPT  time 
sharing  system. 


5.3  The  development  costs  for  the  stimulus 
technique  were  much  lower.  The  analytical 
technique  provided  data  on  the  more  subtle 
behavior  and  complex  interactions  and 
provided  measures  of  performance  on  any 
portion  of  the  system.  The  stimulus 
approach  produced  gross  data  on  the 
behavior  of  the  system  in  terms  cf 
throughput  and  response  time.  The  analytic 
measurement  required  extensive  offline 
reduction  and  analysis  before  the  data  were 
meaningful.  The  measurements  for  the 
stimulus  techniques  were  simple  and 
immediately  obtainable. 

5.4  A  number  of  areas  for  further  development 
for  both  approaches  were  mentioned. 
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1.3  Models  are  constructed  for  various  purposes 
and  with  various  goals  in  mind.  Four 
general  functions  of  models  are: 

(1)  To  formalize  scientific  theory. 

(2)  To  improve  man-machine  theory. 
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(3)  To  improve  automatic  devices. 

(4)  To  measure  "hidden"  processes. 

Some  models  are  built  solely  to  replace  man 
in  specific  tasks,  and  should  be  judged 
solely  on  how  well  they  perform  these 
tasks,  regardless  of  whether  they  perform 
them  like  the  man  does.  But  the  more 
common  purpose  of  the  model  is  to  help 
understand  human  performance.  In  that 
case,  four  criteria  for  Judging  the  worth 
of  a  model  are  proposed: 

(1)  Accuracy — Does  the  model's  performance 
of  a  task  produce  a  response  record 
closely  resembling  that  of  a  human 
operator? 

(2)  Verifiability — Are  the  processes  that 
underlie  the  model's  performance 
similar  to  ti.ose  that  underlie  human 
performance? 

(3)  Generality — Is  the  model  applicable  to 
a  large  range  of  tasks,  or  only  to  a 
few? 


6.  Further  Research  Areas 

6.1  Measurement  System 
Limitations 

6.2  Research  Potentials/ 
Priorities 

6.3  Research  Planning 
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(4)  Parsimony — Is  the  model  unnecessarily 
complex? 

3.5  Th»  author  reviews  five  classes  of  human 

operator  models  in  more  or  less  historical 
sequence: 

(1)  Continuous  transfer  and  describing 
function  models. 

(2)  Operator  intermitteney:  Sampled  data 
models. 

(3)  Operator  non-linearities:  Some 
partial  models. 

(4)  Computer  analogs. 

(5)  Predicted  and  preview  models. 

6.1  The  point  is  made  that  human  performance  is 
not  linear,  and  may  be  poorly  represented 
by  linear  control-theory  models,  except  for 
certain  fairly  simple  or  restricted  tasks. 
Also,  human  control  is  exercised  not  on  the 
basis  of  present  error,  but  rather  on  the 
basis  of  predicted  future  error. 
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A  comparison  of  Register  Transfer  (RT) 
level  modeling  and  V-PMS  is  quite 
indicative.  While  RT-jnodeling  approaches 
a  restricted  goal,  viz.  a  hardware 
structure  capable  of  performing  a  few 
hundred  algorithms,  a  V-PMS  level  model  has 
a  complete  computer  system  as  its  target, 
i.e.  a  composite  of  hardware  structures  of 
the  RT-level  complexity  cooperating  under 
the  control  of  an  operating  system  in  the 
execution  of  a  load. 

In  order  to  construct  a  language  suitable 
for  describing  the  hardware  as  part  of  a 
total  PMS-level  model  for  performance 
evaluation,  the  original  form  of  FMS  was 
substantially  changed  by  providing  an 
expanded  set  of  building  blocks  with 
corresponding  definitions.  This  language, 
with  its  clearly  defined  functions  and 
performance  data,  its  unambiguous  communica¬ 
tion  blocks  and  rules  for  interconnections, 
provides  a  human  reader  with  a  clear  under¬ 
standing  of  the  performance  of  components 
and  of  their  internal  communication  within 
the  computer  system,  and  links  the  hardware 
part  to  a  model  of  an  operating  system  (to 
be  supplied  at  a  later  time).  For  computer 
readable  system  specification  a  syntax  for 
connecting  the  above  symbolic  components  is 
proposed .  A  description  of  the  CDC  Cyber 
74/CDC  6600  system  examplifies  the  use  of 
the  proposed  language  and  its  merits  for 
building  performance  evaluation  models. 
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This  report  discusses  the  test  of  a 
"man-in-the-loop"  program  conducted  with  an 
aircrew  oxygen  system. 

The  system  was  the  NAOS  Flight  Breadboard 
System  which  uses  electrochemical  generator 
and  carbon  dioxide  removal.  Human  and 
animal  test  subjects  participated  in  this 
study. 

The  mission  of  this  system  was  to  provide 
aircrews  with  a  safe,  reliable,  and  compact 
oxygen  system. 

Two  experiments,  one  involving  animals  and 
one  involving  human  test  subjects,  were 
conducted  in  a  laboratory  situation. 

The  ultimate  performance  requirement  of 
this  system  was  that  aircrews  receive  the 
life  support  necessary  during  flight. 

The  object  of  the  study  was  to  provide  a 
safe,  reliable,  compact  system  which  would 
replace  present  systems  minimizing  the  need 
for  ground  support  facilities  and  reduced 
time  and  effort  for  servicing. 

Two  experiments  were  conducted  in  this 
study.  The  first  experiment  was  conducted 
to  measure  the  effects  of  oxygen  generated 
by  the  Flight  Breadboard  System  on  lung 
tissue  of  small  animals.  The  second 
experiment  involving  human  test  subjects 
concerned  the  physiological  effects  and 
comfort  of  the  equipment. 
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The  specific  performance  requirement  of 
this  system  was  that  the  system  meet  safety 
and  comfort  limits. 

Subjective  judgments  of  comfort  were 
obtained  when  evaluating  different  masks 
and  suspension  systems.  Determination  of 
the  physiological  aspects  were  obtained  by 
physical  examination  of  the  test  subjects 
and  based  on  expert  judgments. 

The  animal  tests  were  conducted  in  a 
plexiglass  enclosure  through  which 
rebreather  gases  were  calculated.  Human 
tests  utilized  aviation  oxygen  masks  and 
the  Flight  Breadboard  System. 

Hamsters  and  mice  participated  in  the 
animal  experiment.  Four  members  of  the 
project  team  volunteered  for  the  human 
tests. 


The  animals  received  a  single  acute 
exposure  of  3.5  hours  duration  and  a 
chronic  exposure  of  5.5  hours/day  for  ten 
consecutive  days  of  rebreather  gases.  The 
animals  were  sacrificed  and  lab  examina¬ 
tions  conducted  on  lung  tissue. 

In  the  human  tests,  two  series  were 
conducted.  In  the  first  test,  subjects 
experienced  the  system  operated  with  and 
without  safety  presssure  to  determine 
comfort  levels  and  possible  physiological 
damage  to  respiratory  systems.  The  second 
test  was  to  determine  relative  comfort  of 
alternative  equipment.  In  both  tests 
carbon  dioxide  levels  and  oxygen  levels 
were  monitored.  Mask  leakage  measurements 
were  also  made  in  relation  to  the  employ¬ 
ment  of  safety  pressure  and  comfort  levels. 

The  system  operated  successfully  in  most 
areas  of  expected  performance.  However, 
mask  leakage  problems  must  be  solved  if  the 
full  advantage  of  closed-loop  oxygen 
systems  is  to  be  realized.  It  is 
recommended  that  future  testing  make 
provisions  for  a  higher  oxygen  rate  to  be 
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This  report  addresses  the  integrated 
operational  test  and  analysis  procedures 
for  small  arms  weapon  system  evaluation. 

The  system  was  composed  of  infantrymen, 
their  weapons,  and  equipment. 

The  mission  was  the  use  of  small  arms  in 
combat  situations  by  infantrymen. 

Simulated  combat  conditions  were  utilized 
in  this  study. 

The  ability  to  duplicate  combat  actions  and 
tasks  in  a  test  facility  affected  the 
validity  of  test  results. 

The  ultimate  performance  requirement  of 
this  system  is  mission  accomplishment  in 
which  the  mission  is  to  close  with  and 
defeat  the  enemy. 

One  of  the  primary  measures  of  success  of  a 
combat  mission  was  described  as  the  number 
of  enemy  casualties. 

The  performance  of  infantrymen  using  small 
arms  weapons  was  evaluated.  The  criteria 
used  were  accuracy,  sustainability, 
responsiveness,  and  reliability. 

The  number  of  hits  was  the  measure  of 
effectiveness  of  this  system. 

The  criteria  for  the  evaluation  of  weapon 
system  performance  fell  into  four 
categories: 
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(1)  Accuracy  (number  of  hits).,  hit 
probability,  first  round  hit 
probability,  engagement  hit 
probability,  distribution  of  near 
misses. 

(2)  Sustainability  (number  of  hits  per 
round  and  hits  per  basic  load). 

(3)  Responsiveness  (time  to  fire  first 
round,  time  to  first  hit,  time  betwei 
rounds,  time  to  shift  fire). 

(4)  Reliability  (number  of  malfunctions, 
numbers  of  rounds  between 
malfunctions,  and  time  to  clear 
malfunctions) . 

3.5  For  the  purpose  of  this  study,  the  missioi 
accomplishment  measure  was  equal  to  the 
number  of  target  hits. 

4.1  Sample  size  should  be  based  on  the  specif 
test  criteria  and  selection  of  the  appropi 
ate  MOE.  The  technique  for  performing  th 
primary  analysis  is  a  3x2x2  factorial  expi 
iment.  The  factors  were  facilities,  weap¬ 
ons  and  modes  of  fire.  A  linear  model  cai 
be  developed  and  an  analysis  of  variance 
can  be  performed. 

4.2  The  sample  of  test  soldiers  should  be 
representative  of  the  infantry  as  a  whole 
in  terms  of  age,  rank,  experience  and 
physical  attributes.  Weaptn  should  be 
assigned  on  the  basis  of  a  performance 
measure.  Training  was  found  to  be  a 
substantial  source  of  bias.  An  accelerati 
training  class  was  held  to  familiarize  th< 
subject  with  new  weapons.  Results  showed 
bias  favoring  a  standard  weapon  but  time 
limitation  prevented  elimination  of  this 
bias.  A  balanced  experimental  design  was 
implemented.  The  test  facility  was 
designed  to  duplicate  as  closely  as 
possible  the  "real-world"  combat-situatioi 
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i|.4  Ninety-six  infantrymen/in  96  rifle  systems 
participated  in  this  experiment. 

M.5  The  components  of  the  test  plan  were: 

selection  of  subjects,  determination  of 
sample  size,  weapon  assignment,  training  of 
subjects,  scheduling,  test  facility 
determinants,  test  implementation,  and  data 
analysis. 
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This  paper  discusses  the  development  of 
combat  related  measures  for  small  arms 
evaluation. 

The  system  studied  included  the  infantry 
soldier,  his  weapons,  and  equipment. 

The  mission  of  the  system  was  the  most 
effective  use  of  men,  equipment,  and 
weapons  in  a  combat  situation. 

The  specific  requirements  of  the  system  was 
the  achievement  of  a  "hit"  during  a 
quick-fire  engagement  in  the  shortest 
period  of  time. 

A  list  of  26  separate  combat-actions  was 
prepared.  Secondly,  a  list  of  tasks 
normally  accomplished  by  the  infantryman 
when  executing  the  combat  actions  was 
developed.  It  was  determined  that  three 
basic  tactical  situations  (attack, 
quickfire,  and  defense)  would  accommodate 
all  these  actions  and  tasks. 

Twenty-six  measures  of  effectiveness  were 
developed.  These  included  time  to  first 
round,  time  between  trigger  pulls, 
distribution  of  near  misses,  time  to  shift 
fire,  and  hits  per  pound  expressed  as  a 
percent  of  a  soldier's  basic  load. 

Several  measurement  procedures  were  used: 
the  time  to  fire  first  roi  .xi  measures,  the 
time  it  takes  for  a  soldier  carrying  his 
rifle  at  ready  position  to  engage  a 
surprise  target,  and  the  time  between 
trigger  pulls  which  is  defined  as  an 
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indicator  of  the  soldier’s  ability  to 
absorb  the  recoil,  acquire  the  target, 
obtain  a  new  sight-picture,  point  the 
weapon  and  squeeze  the  trigger.  An 
effectiveness  scale  of  five  levels  was 
developed . 

4.1  The  firing  engagements  were  categorized 

according  to  the  degrees  of  effectiveness 
as  defined  by  whether  or  not  the  firer 
achieved  a  hit.  Those  engagements  which 
required  multiple  trigger  pulls  and  failed 
to  achieve  a  hit  were  defined  as  least 
effective.  Those  which  resulted  in  a  hit, 
whether  single  or  multiple  trigger  pulls 
were  required,  were  designated  as  the  most 
effective.  Time-of-first  round,  target 
range,  time  between  bursts,  time  between 
trigger  pulls,  number  of  rounds  to  first- 
hit,  burst  size,  and  burst  hit  probability 
were  presented  graphically,  and  provided  a 
comparison  between  the  weapons  being 
tested. 

4.3  The  test  equipment  were  two  different 
automatic  rifles,  tested  in  a  simulated 
combat-firing  facility. 

5.3  It  was  determined  that  the  potential  of  one 
weapon  was  greater  than  the  other  being 
tested.  Two  findings  were  considered 
important:  (1)  the  service  test  must 
determine  the  weapons  systems  optimum 
operating  mode  to  yield  complete  informa¬ 
tion  on  weapon  potential  and  (2)  training 
procedures  are  related  directly  to  weapon 
performance  and  therefore  should  not  be 
considered  separate  entities. 

5.4  It  is  felt  that  this  technique  of  equating 
a  specific  measure  to  real  world  effective¬ 
ness  represents  an  advance  in  military  test 
procedures,  but  is  limited  in  that  the 
"real  world”  is  still  a  simulated  combat 
firing  facility.  However,  with  these 
limitations  the  facility  is  a  dynamic  test 
environment  and  brings  into  play  many  of 
the  influencing  variables  common  to  the 
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combat  environment  and  provides  an 
improvement  in  operational  testing.  The 
ultimate  value  to  be  realized  from  this 
methodology  is  how  well  these  statistical 
measures  can  be  weighed  in  this 
"real-world"  atmosphere. 
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Optimal  control  and  estimation  theory  may 
be  used  to  develop  a  model  of  human 
response  in  manual  tracking  tasks.  The 
tasks  considered  were  those  in  which  the 
controlled  element  was  linear  and  the 
system  was  disturbed  by  a  white  noise 
input.  The  model  included  representations 
of  human  limitations  and  a  cascade 
combination  of  a  Kalman  filter,  a 
least-mean-squared  predictor,  and  a  set  of 
optimal  feedback  gains  as  compensating 
elements.  An  "optimal  scanning  mechanism" 
was  also  added  to  the  model  to  account  for 
situations  where  the  human  operator  must 
visually  scan  several  instruments  in  order 
to  achieve  his  control  objectives. 

The  use  of  the  model  in  predicting  task 
performance,  controller  describing 
functions,  and  power  spectra  was 
demonstrated.  The  model  was  then  validated 
by  comparing  model  results  with  experimen¬ 
tal  data  from  three  simple,  but  classical, 
manual  control  tasks. 
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Sensitivity  studies  of  simple  manual 
control  tasks  (i.e.,  single  input,  single 
display  indicator)  were  conducted.  The 
measures  examined  incuded: 

(1)  Neuromotor  time  constant. 

(2)  Time  delays. 

(3)  Observation  noise. 

(4)  Motor  noise. 
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A  prediction  of  pilot  performance  in  a 
hovering  task  was  also  attempted. 

Agreement  between  measured  and  predicted 
quantities  was  obtained,  demonstrating  the 
value  and  potential  of  the  optimization 
approach  to  manned-vehicle  systems 
analysis. 
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The  purpose  of  this  study  was  to  review 
existing  human  modeling  techniques  and 
evaluate  their  potential  utility  for 
performance  measurement  applications.  It 
appeared  that  existing  performance 
measurement  techniques  do  not  have  the 
capability  to  support  the  type  of  flight 
simulation  research  that  entails  accounting 
for  the  perception  and  utilization  of 
various  cues. 

Since  model  validity  is  particularly 
important  in  the  case  of  the  envisioned 
measurement  applications,  the  first  task 
was  to  identify  the  major  human  operator 
characteristics  that  ought  to  be  accounted 
for . 

A  review  of  work  dating  from  1944  was 
conducted  and  this  material  was  categorized 
into  six  types:  (1)  describing  functions, 
(2)  optimal  control  model,  (3)  discrete 
and  finite  state  methods,  (4)  adaptive 
techniques,  (5)  preview  models,  (6)  other 
nonlinear  approaches.  A  survey  of  models 
in  each  category  was  made  by  reviewing  the 
literature  and  summarizing  various  modeling 
studies.  Models  in  each  category  were 
evaluated  based  on  the  extent  to  which  they 
represented  the  identified  human  operator 
characteristics  as  well  as  other  aspects  of 
their  general  validity  for  performance 
measurement  applications. 

Results  show  that  none  of  the  models 
implemented  more  than  a  few  of  the  operator 
characteristics;  many  are  based  on 
assumptions  which  are  unacceptable  for 
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measurement  applications  and  others  have 
not  been  developed  far  enough  to  justify 
their  use  as  a  po.nt  of  departure  for 
measurements . 


It  is  concluded  that  existing  models  are 
not  sufficiently  representative  of  known 
characteristics  to  be  useful  for  general 
applications  in  performance  measurements 


Knowles,  W.B.,  Burger,  W.J.,  Mitchell,  M.B.,  Hanifan,  D.T.  4  Wulfeck,  J.W. 
Models,  measures,  and  judgments  in  system  design.  Human  Factors,  1969,  11. 
577-590  ” 
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Current  practice  of  system  design  commonly 
requires  the  use  of  analytical  models. 

While  such  tools  are  finding  increasingly 
wide  application  and  acceptance,  their  use 
is  often  limited  by  the  lack  of  data  in 
forms  suitable  for  modeling  effects. 
Overall,  there  can  be  little  doubt  that  the 
design  of  systems  would  be  greatly 
facilitated  if  tasks  and  equipment  could  be 
treated  within  the  same  conceptual  scheme. 
Unfortunately,  techniques  originally 
developed  to  describe  and  predict  human 
performance  for  that  purpose  have  fallen 
short  of  the  goals. 

The  following  approach  is  suggested  for 
studying  system  design: 

(1)  Identify  a  practical  situation  where 
suitable  populations  of  expert  judges 
and  tasks  can  be  obtained,  and  where 
estimates  and  actual  measures  of  task 
performance  can  be  obtained. 

(2)  Conduct  a  study  to  identify  the 
relevant  dimensions  along  which  to 
categorize  tasks  and  judges,  and  to 
obtain  correlations  between  estimated 
and  actual  performance. 

(3)  Develop  means  for  structuring  the 
presentation  of  stimulus  materials, 
i.e.,  task  descriptions,  so  that  the 
relevant  aspects  of  the  tasks  are 
suitably  displayed. 


6.  Further  Research  Areas 

6.1  Measurement  System 
Limitations 

6.2  Research  Potentials/ 
Priorities 

6.3  Research  Planning 
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(4)  Develop  means  for  classifying  judges 
and  for  selecting  "valid"  individual 
judges. 

(5)  Conduct  a  study  to  revalidate  the 
estimates  obtained  with  the  improved 
methods. 

(6)  Develop  category  scaling  procedures  to 
be  used  generally  in  evaluating 
equipment-oriented  tasks,  and  develop 
a  set  of  standard  stimulus  materials 
to  be  used  in  differentiating  among 
prospective  judges. 


Kreifeldt,  J.  Analysis  of  predictor  model.  Unpublished  Preliminary  Data, 
Massachusetts  Institute  of  Technology,  1954, 
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1.1  This  memorandum  discusses  an  analysis  of  a 
predictor  model  which  realistically 
accounts  for  inputs  which  have  spatial  as 
well  as  time  dependence.  An  analysis  is 
presented  which  attempts  to  abstract  the 
real  world  to  some  degree. 
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In  this  outline  a  model  of  a  human  operator 
controlling  a  vehicle  was  analyzed.  The 
model  attempts  to  account  for  the  fact  that 
in  many  situations  (i.e.,  driving)  the 
operator  has  an  input  which  is  not  a  single 
point  in  time  but  an  input  which  has 
spatial  as  well  as  time  features,  that  is, 
he  can  look  at  the  road  ahead. 


3.  Analytic  Components 
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The  sampling  theorem  in  spatial  coordinates 
was  invoked  in  order  to  treat  the 
time-space  input  as  k  discrete  inputs  to 
the  operator  simultaneously  available.  The 
model  then  states  that  the  operator  runs 
some  sort  of  thought  experiment  in  which 
position  is  extrapolated  and  future  error 
is  computed  if  the  same  control  signal  is 
maintained.  This  computed  and  weighted 
predicted  error  forms  the  basis  for  the 
operator's  control  action. 

The  model  performs  the  thought  experiment 
by  computing  from  vehicle  initial 
conditions  and  command  signal  what  the 
errors  will  be  at  the  previewed  points. 
These  individual  errors  are  simultaneously 
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computed  and  weighted  in  a  length  of  time 
requiring  tt  seconds. 

The  transformed  impulse  response  for  this 
model  was  derived  and  composed  of  discrete 
and  continuous  elements. 

This  impulse  response  was  specifically 
evaluated  for  a  first-order  vehicle  and  two 
input  points.  It  was  seen  that  if  stable, 
it  eventually  reaches  a  reference  step 
height  input. 


Kurke,  M.I.  Operational  concept  analysis  and  sources  of  field  data. 
Human  Factors,  1965,  (7),  537-5M. 
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The  human  factors  approach  to  an  operational 
analysis  should  be  commensurate  with  that 
of  the  operations  research  approach.  One 
approach  to  this  methodology  is  described 
in  this  paper.  A  hypothetical  example  (i.e., 
a  rock)  is  given  in  which  the  essential 
elements  of  analysis  are  presented  in  order 
to  determine  whether  an  item  of  materiel 
should  be  included  in  an  Army  inventory. 

The  five  major  functions  described  are: 
firepower,  mobility,  combat  service  support, 
intelligence  and  reconnaissance,  command, 
control  and  communications. 

The  functions  are  analyzed  into  subfunctions 
which  are  relevant  until  a  level  of  descrip¬ 
tion  is  reached  that  can  be  measured  opera¬ 
tionally.  For  example,  firepower  can  be  a 
measure  of  hit  probability  as  a  function  of 
range.  In  addition,  if  is  determined  from 
an  analysis  of  the  subfunction,  mobility, 
that  the  ability  to  throw  the  hypothetical 
weapon  (i.e.,  the  rock)  while  running  is  a 
very  important  operational  capability,  then 
this  ability  would  be  an  attribute  measured 
in  terras  of  accuracy  when  using  protective 
equipment  versus  not  using  protective 
equipment. 

The  essential  elements  of  the  analysis 
include:  effectiveness  in  attaining  objec¬ 
tives,  effectiveness  when  chemical  agent 
protective  equipment  is  being  used,  training 
requirements,  and  operating  costs. 

The  field  data  collected  for  military  OR 
studies  come  from  three  sources:  field  exer¬ 
cises,  troop  tests,  and  field  experiment* . 
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The  characteristics  of  these  three  sources 
of  field  data  for  operational  concept  and 
evaluation  have  been  compared  with  the 
field  data  sources  for  systems  analysis 
and  evaluation. 

The  three  sources  of  field  data  are  defined 
as  follows: 

(1)  Field  Exercise — an  exercise  conducted 
in  the  field,  under  simulated  war 
conditions,  in  which  troops  and  arma¬ 
ments  of  one  side  are  actually  present, 
while  the  other  side  may  be  imaginary 
or  in  outline. 

(2)  Field  Experiment — an  investigation  to 
experiment  with  or  evaluate  new  or 
revised  doctrine  and  organizations, 
and  new,  modified  or  current  materiel 
in  order  to  develop  combat  capabilities 

(3)  Combat  Development  Troop  Tests — a  field 
investigation  designed  to  test  the 
ability  of  a  prototype  organizational 
structure  to  follow  a  specific  doctrine 
using  specific  equipment  to  complete  a 
specific  mission  and/or  to  test  the 
concept  of  operations  as  limited  by  the 
structure  and  functions  of  a  prototype 
organization. 

A  table  is  presented  which  compares  the 
three  sources  of  army  field  data  in  terms 
of  methods,  kinds  of  analyses,  reliability 
of  data  and  types  and  usefulness  of  results. 
In  a  study  of  32  tests  and  experiments  pub¬ 
lished  in  1963,  it  was  noted  that  only  the 
field  experiments  and  troop  tests  provided 
systematic  data. 

The  field  experiments  tended  to  gather  base 
data,  compare  systems  and  collect  informa¬ 
tion  about  specific  variables.  However, 
although  more  than  half  of  the  conclusions 
in  these  experiments  were  drawn  from  objec¬ 
tive  data,  the  balance  supplemented  their 
results  with  subjective  data. 
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Some  troop  tests  were  developed  to  collect 
base  data  about  a  system  but  only  one  out 
of  eight  compared  systems.  Generally,  in 
these  reports,  test  objectives  were  not 
clearly  stated.  Few  of  these  tests  investi¬ 
gated  specific  variables  although  all  in¬ 
vestigated  the  total  system.  All  troop 
test  conclusions  were  based  on  subjective 
data.  More  than  half  of  these  troop  tests 
were  uncontrolled  both  statistically  and 
experimentally  and  none  presented  anything 
more  sophisticated  than  a  measure  of 
control  tendency. 

The  test  plan  for  functional  analysis  would 
include: 

(1)  Development  of  a  frame  of  reference 
for  operations. 

(2)  Determination  of  the  functions  or  sub¬ 
functions  relevant  to  describing 
effectiveness.  In  this  analysis,  some 
screening  or  weighting  of  functions  in 
terms  of  their  relative  importance  may 
be  performed . 

(3)  Development  of  operational  criteria 
based  on  the  determination  of  relevant 
functions. 

(4)  Refinement  of  the  essential  elements  of 
the  analysis  in  terms  of  specific  per¬ 
formance  criteria. 

(5)  Determination  of  the  appropriate  opera¬ 
tional  criteria,  their  measures  and 
the  correct  mode  of  collecting  the 
information. 

(5)  Determination  of  appropriate  data 
collection  techniques,  i.e.,  field 
experiment  supplemented  y  computer 
simulations. 
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1.3  The  present  document  presents  a  theoretical 
model  for  the  characterization  of  the 
sampling  behavior  of  an  optimal  process 
monitor.  The  dynamic  decision  model  was 
based  on  the  proposition  that  an  ideal 
monitor  adopts  a  sampling  strategy  that,  at 
any  point,  maximizes  some  expected  value 
(worth  or  utility)  for  the  monitoring  of  a 
single  signal  with  a  fixed  cost  of  sampling 
and  of  the  signal  exceeding  certain 
threshold  units.  The  present  model  was 
shown  to  lead  to  the  strategy  that  sampling 
should  occur  when  the  conditional 
probability  of  the  signal  going  beyond  the 
threshold  units  exceeds  the  ratio  of  the 
sampling  cost  to  the  threshold  cost. 
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2.1  The  St.  Louis  [Railroad]  Terminal  Project 
is  an  experimental  program  seeking 
ultimately  to  produce  a  more  saleable 
transportation  service  and  greater  profit 
to  the  railroads.  The  experimental 
procedure  places  heavy  emphasis  on 
quantitative  measurement  of  the  impact  of 
operational  changes.  To  determine  the  best 
method  for  measuring  performance  of  the 
terminal  as  a  whole,  a  seminar  on  terminal 
performance  measurement  systems  was 
conducted  by  the  St.  Louis  Project  Team. 

2.2  Current  terminal  performance  measurement 
systems  were  found  to  be  designed  to 
support  one  or  more  of  the  following 
functions: 

(1)  Evaluate  performance  and  trigger  the 
planning  process  to  develop  changes 
that  will  produce  improved 
performance. 

(2)  Evaluate  experimental  changes  in 
operations  to  determine  the  actual 
improvement  in  performance. 

(3)  Monitor  the  operations  to  provide 
information  that  results  in  corrective 
action  to  prevent  a  deterioration  in 
performance. 

(4)  Assess  the  performance  of  the  managers 
responsible  for  the  operations. 
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3.1  Observations  of  the  various  terminal 

performance  measurement  systems  indicated 
that: 


(1)  All  measurements  systems  use  car 
movement  data  collected  by  real-time 
[automated]  systems,  with  no 
additional  burden  placed  on  clerical 
forces. 

(2)  Speed  of  car  movements  is  mersured  by 
the  time  cars  spend  under  a  terminal 
superintendent's  responsibility. 

(3)  Reliability  of  car  movements  is 
measured  by  comparing  elapsed  time 
with  a  standard. 

(4)  Most  systems  measure  the  car  time 
costs  of  car  movements. 

(5)  The  systems  currently  in  use  do  not 
measure  all  of  the  variables  with 
which  terminal  management  must  deal. 


Larson,  O.A.,  Sander,  S.I.  A  Steinemann,  J.H.  Survey  of  unit  performance 
effectiveness  measures  (TR74-11).  San  Diego,  CA:  Navy  Personnel  Research  and 
Development  Center,  January  1 97^ .  (AD-774  919). 
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A  survey  to  determine  the  state  of  the  art 
of  performance  assessment  systems  and 
methodologies  was  conducted  to  support  a 
Marine  Corps  requirement  for  improved 
measures  of  performance  effectiveness  to  be 
used  in  conjunction  with  the  Tactical 
Warfare  Analysis  and  Evaluation  System 
(TWAES) . 

In  addition  to  a  literature  search, 
researchers  observed  field  exercises  and 
examined  related  U.S.  Army  systems. 

A  number  of  areas  were  identified  as 
needing  support  efforts  including  timely 
reporting  and  assessment  of  events  by 
umpire  personnel. 

It  was  concluded  that  the  literature  does 
not  contain  substantive  research  studies 
defining  human  performance  assessment 
methods  and  criteria  suitable  for  direct 
application  to  TWAES.  Studies  were  found, 
however,  that  will  be  of  use  in  initiating 
and  structuring  the  research  in  this  area. 
Organizational,  situational  and  evaluative 
factors  all  hold  promise  as  potential 
sources  of  performance  effectiveness 
measures  along  with  the  use  of  Delphi-type 
approaches  for  the  development  of  interim 
methods  and  criteria. 


Lawrence,  D.H.  The  evaluation  of  training  and  transfer  programs  in  terms  of 
efficiency  measures.  The  Journal  of  Psychology,  1954,  36,  367-382.  (AD-87 
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This  paper  discusses  the  decision  making 
involved  in  designing  training  programs. 

It  is  noted  that  programs  are  designed  to 
produce  the  desired  level  of  proficiency 
with  the  minimum  outlay  of  time,  energy  and 
cost.  However,  it  is  theorized  here  that 
the  implications  of  such  a  principle  are 
not  explicitly  formulated  as  they  bear  on 
the  efficacy  of  the  decisions  or  on  the 
research  needed  as  a  basis  for  such 
decision . 

This  paper  attempts  to  remedy  part  of  this 
deficiency  by  means  of  a  preliminary, 
simplified  simulation  of  the  training  and 
transfer  problem  in  terms  of  the  principle 
of  minimization  of  expenditure. 

Simplified  assumptions  are  made  about  cost 
functions  and  learning  curves  involved  in 
training  to  permit  a  clear  expression  of 
the  methodology  involved.  Any  use  of  this 
methodology  in  other  specific  training 
situations  may  involve  a  different  set  of 
assumptions. 

A  mathematical  formula  is  presented  which 
demonstrates  a  relationship  between  the 
cost  and  the  stage  of  training.  This 
states  that,  in  general  cost  is  universally 
related  to  the  level  of  proficiency 
achieved  at  any  given  time.  Factors 
influencing  the  cost  of  training  an 
individual  are  considered  as  well  as  the 
average  cost  for  group  training. 

The  paper  discusses  ways  in  which  training 
costs  can  be  reduced  by  means  of  improved 
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training  techniques,  selection  of 
individuals  to  be  trained,  and  by  the 
utilization  of  the  transfer  of  training 
procedures. 

It  is  pointed  out  that  standardized 
training  programs  which  bring  all 
individuals  to  a  level  of  proficiency  can 
result  in  overtraining  for  some.  Cost 
benefits  of  considering  individual 
differences  by  modifying  training  programs 
are  discussed. 

The  final  section  of  this  paper  deals  with 
the  employment  of  transfer  of  training 
situations  as  a  means  of  reducing  training 
costs.  The  assumption  underlying  the 
transfer  paradigm  is  that  an  individual  can 
be  trained  to  a  desired  level  of  perform¬ 
ance  easier  and  with  less  cost  if  he  is 
first  trained  on  a  preliminary  task. 


Levy,  G.W.  Criteria  for  selection  and  application  of  models.  In  G.W.  Levy 
(Ed.),  Symposium  on  applied  model  of  man-machine  systems  performance 
(NR69H-591 ) .  Columbus,  OH:  North  American  Aviation,  November  19 >8.  (AD-697 
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1.3  This  paper  presents  some  thoughts  on 

considerations  for  selecting  and  applying 
man-machine  system  models.  The  author 
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(2)  Those  relating  to  the  precision  and 
validity  of  the  model  predictions. 

Concerning  the  model's  output,  considera¬ 
tion  should  be  given  to  whether  the  model 
predicts  human  performance  or  system 
performance.  System  performance  measures 
involve  parameters  of  both  human  and 
machine  and  even  their  interaction.  Such 
measures  imply  a  system  effectiveness  mode 
of  thought,  a  concern  with  overall  system 
performance  rather  than  human  performance 
per  se.  However,  it  becomes  necessary  to 
separate  man  and  machine  performance 
explicitly  and  to  develop  human  performance 
submodels  that  include  system-oriented 
human  performance  measures.  Examples  of 
these  may  include  mean  time  to  repair, 
slant  range  at  target  identification,  etc. 

As  regards  model  input,  an  initial 
consideration  should  be  the  number  of 
variables  involved.  Often,  too  many 
variables  are  specified,  either  because  an 
attempt  is  made  to  achieve  an  unrealistic 
predictive  precision,  or  because  it  isn't 
possible  to  separate  meaningful  variables 
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6.1  Measurement  System 
Limitations 

6.2  Research  Potentials/ 
Priorities 

6.3  Research  Planning 
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from  the  great  number  of  possible 
variables.  On  the  other  hand,  meaningful 
variables  sometimes  are  overlooked  because 
they  haven't  been  studied  empirically. 
Operator  motivation  is  an  example  of  a 
variable  that  most  models  disregard.  A 
second  consideration  related  to  input  is 
the  existence  of  the  necessary  data. 
Thirdly,  model  parameters  may  also  be 
treated  as  needed  input  for  application  of 
the  model:  the  significant  consideration 
is  whether  these  parameters  are  fixed 
beforehand  or  estimated  from  the  data. 

Concerning  the  form  of  the  model,  the  first 
issue  is  whether  it  is  an  empirical  or  a 
theoretical  model.  Theoretical  models  are 
obtained  by  a  hypothetical-deductive 
procedure,  empirical  model  equations  are 
obtained  by  curve  fitting.  Theoretical 
models  provide  a  guide  to  research  and  a 
programmaticity  which  empirical  models 
lack,  but  theoretical  models  involve 
assumptions  which  may  or  may  not  have  been 
tested.  Another  issue  concerning  form  is 
"what  kind  of  man  is  being  modeled."  Is  it 
an  individual  with  individualistic 
parameters  or  is  it  the  famous  average  man? 
Is  he  trained  or  untrained?  Few  models 
handle  the  problems  of  individual 
differences  and  training. 

5.3  Concerning  the  precision  or  accuracy 

required  of  a  model,  this  is  generally 
regarded  to  be  a  function  of  the  stage  of 
the  system  life  cycle  in  which  the  model  is 
being  used.  However,  it  is  believed  that 
the  same  levels  of  precision  are  required 
in  the  initital  stages  of  design  as  in 
later  applications.  In  order  to  be  truly 
useful,  applied  models  must  consider  the 
relevant  interactions  of  design  parameters 
with  difficulty  of  conditions  and  must 
possess  a  sufficient  degree  of  accuracy. 
Unfortunately,  the  criterion  of  how  much 
accuracy  has  never  been  spelled  out. 
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6.1  Turning  to  validity,  it  is  noted  that  the 
vast  majority  of  man-machine  system  models 
have  never  been  validated.  It  is  suggested 
that  a  research  design  termed  the  "field 
laboratory  validation  design"  is  very  useful 
in  developing  and  validating  applied 
models.  This  design  calls  for  collection 
of  performance  data  and  input  data  in  field 
situations,  with  the  input  data  recorded 
for  use  in  laboratory  studies  aimed  at 
model  development.  The  models  in  turn  are 
validated  by  comparing  their  outputs  with 
the  pre-collected  field  performance  data. 


Lindsey,  M.B.  A  human  performance  investigation  in  a  NAVORD  weapon  system  - 
Part  2  (Report  27-349).  Annapolis,  MD:  Naval  Ship  Research  and  Development 
Center,  Propulsion  and  Auxiliary  Systems  Dept.,  March  1974.  (AD-918  202L) 
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This  study  is  part  II  of  an  investigation 
regarding  integration  of  ordnance  systems 
personnel  functions  and  the  design  of 
man-made  interfaces.  A  specific 
modification  of  a  methodology  was  developed 
to  yield  quantitative  data  for  the  purpose 
of  determining  the  performance  levels  of 
maintainers/operators  in  ASROC 
(Antisubmarine  Rocket)  launching  groups. 

If  "performance"  means  demonstrable 
operational  capability,  then  a  system's 
performance  is  taken  as  the  total  behavior 
of  its  man  and  machine  module  components 
interacting  over  a  specified  period  of  time 
in  the  environment  for  which  it  was 
designed;  its  reliability  is  the 
probability  of  performance. 

Previous  researchers  have  categorized  the 
major  variables  that  affect  system 
performance  as; 

(1)  Personnel 

(2)  Equipment 

(3)  Procedure 

(4)  Environment 

(5)  Administration 

(6)  Interaction 

The  statistical  methods  used  to  determine 
the  performance  levels  were  calculations  of 
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the  mean  (including  the  grand  mean),  the 
standard  deviation,  and  the  range. 

A  job  performance  questionnaire  was 
developed  for  the  job  activities  as  a 
cross-section  of  ASROC;  the  work  done  by 
Siegel  on  multidimensional  scaling  was 
found  to  be  applicable  with  only  minor 
revisions  made  (definitions  appropriate  to 
ASROC  were  adjusted  accordingly.) 


I  Topics  Relevant 

1  1 

1  1 

i  to  System  Development 

i Topic! 

land  Evaluation  Technology 

i  No.  ! 

ABSTRACT 

3.1  The  primary  objective  of  the  thermal  model 
is  to  provide  a  tool  for  performance 
predictions  of  the  radiator  system  under 
the  design  conditions  of  maximum  and 
minimum  heat  rejection.  A  single  tube  is 
used  to  model  the  bank  of  11  main  tubes. 

The  resulting  model  thus  consists  of  two 
separate  models  -  one  for  the  mair  system 
for  maximum  heat  rejection  conditions,  one 
for  the  prime  system  for  minimum  heat 
rejection.  There  are  no  thermal 
connections  between  the  two  systems. 

5.2  Comparisons  were  made  between  test  results 
and  (model)  predictions  for  selected  test 
points.  The  model  predictions  agreed  well 
with  the  test  data.  Excellent  agreement 
was  achieved  for  the  high  heat  load 
conditions.  Good  performance  predictions 
also  were  produced  under  minimum  load 
design  conditions,  although  these  correla¬ 
tions  generally  were  not  as  good  as  were 
those  for  high  load. 

In  summary,  a  thermal  model  of  the  proposed 
space  shuttle  modular  radiator  system  has 
been  developed  and  verified  by  comparison 
to  thermal  vacuum  test  data.  Application 
of  the  test  panel  modeling  techniques  to 
the  (actual)  flight  panel  should  prr  ide  an 
accurate  model  for  the  radiator  system 
performance  evaluation. 
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1.1  The  process  of  developing  the  raw  material 
of  human  potential  deserves  a  system  of 
quality  control  at  least  as  carefully 
developed  as  that  applied  to  the 
manufacturing  process.  The  quality  control 
system  must  accomplish  four  major 
objectives: 
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(2) 

2.3 

Environment  Definition 

2.4 

General  Constraints 

2.5 

Performance 

Requirements,  Ultimate 

(3) 

2.6 

Performance 

Criteria,  Ultimate 

(4) 

Quality  assurance. 

Control  of  student  progress. 

Training  program  improvement. 

Training  system  diagnosis  and  change. 
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Quality  assurance  requires  that 
specifications  be  delineated  in  terms  of 
operational  requirements  and  these 
requirements  be  reflected  in  end-of-course 
proficiency  measures. 

Control  of  student  progress  is  accomplished 
by  developing  a  means  of  selecting  and 
organizing  the  learning  experiences  to 
facilitate  achievement  of  objectives.  This 
can  sometimes  be  achieved  by  dividing  the 
training  program  into  modules  each  with  its 
sub-objectives . 

A  systematic  quality  control  process  must 
identify  weaknesses  and  strengths  in  the 
program  by  assessing  and  diagnosing  the 
performance  of  the  trainee. 
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The  essential  elements  of  a  training 
quality  control  system 
are : 


(1)  Training  objectives  (performance 
requirements) . 

(2)  Proficiency  and  diagnostic  measures. 

(3)  Data  reduction  and  analyses. 

(4)  Procedures  for  decision  and  corrective 
actions. 

(5)  Communication  procedures. 

(6)  Managerial  support. 


Harkel,  G.A.  Toward  a  general  methodology  for  systems  evaluation  (352— R— 13) 
State  .College,  PA:  HRB-Singer,  Inc.,  July  1965.  (AD-619  373). 
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This  report  considers  some  of  the  issues 
involved  in  a  general  theory  of  systems 
evaluation.  It  suggests  an  organized  and 
comprehensive  framework  for  approaching  any 
systems  evaluation. 

Definitions  of  systems  are  considered  and 
characterized  as  an  assembly  of 
interrelated  elements  acting  as  an  integral 
ensemble  in  performing  some  functional 
objective. 

The  report  examines  some  of  the  issues 
involved  in  realization  of  practical 
evaluations.  It  discusses  the  need  to 
define  the  problem  —  why  is  the  evaluation 
being  performed,  which  parts  of  systems 
concepts  are  involved,  etc.  It  is  felt 
that  breaking  down  the  problem  into 
smaller,  more  tractable  problems 
facilitates  a  workable  approach  to  any 
evaluation. 

Fundamental  considerations  such  as  criteria 
selection  are  discussed.  The  report 
stresses  the  need  to  select  the  criteria 
with  great  care.  The  underlying  substance 
of  the  evaluation  process  is  measurement 
and  the  key  to  successful  measurement  and 
evaluation  is  to  be  found  in  criteria 
selection. 

It  is  suggested  that  the  development  of 
general  theory  of  systems  evaluation  can  be 
approached  by  identifying  and  defining 
these  elements  which  can  provide  a  basis 
for  the  overall  evaluation  of  any  system. 
These  elements  may  be  described  in  three 
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broad  areas  of  primary  concern  for  systems 
in  general: 

(1)  Systems  structure. 

(2)  Systems  operations. 

(3)  Systems  performance. 

It  is  felt  that  a  valid  approach  is  to 
employ  a  modular  concept  of  systems 
evaluation  addressing  each  basic  factor  as 
a  separate  module.  Each  application  uses 
only  the  modules  most  appropriate  to  that 
system  and  the  results  obtained  from  the 
various  modules  are  integrated 
subjectively. 

Different  modeling  concepts  are  discussed 
in  the  last  section  of  this  report.  It  is 
felt  that  network  models  are  best  suited 
for  structural  aspects,  hierarchical  tree 
models  are  best  for  studying  aggregation  of 
operational  aspects,  and  black  box  models 
provide  the  best  perspective  for  overall 
performance  appraisal. 


Mason,  A.K.  4  Rigney,  J.W.  Toward  a  general  characterization  of  electronic 
troubleshooting.  In  R.E.  Blanchard  4  D.H.  Harris  (Eds.),  Man-machine 
effectiveness  analysis.  Papers  presented  at  The  Human  Factors  Symposium, 
University  of  California  at  Los  Angeles,  June  15,  1967.  (AD-735  718). 
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1.3  Several  factors  underly  the  desire  to 

formulate  a  troubleshooting  processor  model 
accommodating  a  broad  spectrum  of  possible 
specific  procedures:  (1)  such  a  model 
would  unify  the  alternative  ways  of 
characterizing  and  explaining  human 
technician  troubleshooting  behavior;  (2)  it 
would  serve  as  a  vehicle  for  formulating 
the  cost  effectivneess  structure  associated 
with  troubleshooting  electronic  equipment. 

3.2  Ideally,  a  characterization  of  the 

electronic  troubleshooting  process  would  be 
sensitive  to  the  degree  of  automation  in 
the  troubleshooting  tasks.  Thus,  measures 
of  effectiveness  for  particular  equipment 
could  be  generated  across  the  spectrum  of 
alternative  troubleshooting  processors 
(from  the  automated  to  the  manual),  and 
improve  the  sensitivity  of  the 
"maintainability"  component  of  cost 
effectiveness  models. 
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The  aim  of  this  study  was  the  development 
and  test  of  methods  for  evaluating  operator 
performance  during  field  test  of  complex 
man-machine  systems. 

Several  Navy  aircraft  systems  were  examined 
in  detail/and  two  systems  were  selected  for 
detailed  study  —  the  A-7A  and  the  P-3C 
anti-submarine  warfare  systems.  The  major 
portion  of  the  work  was  carried  out  using 
the  P-3C. 

Tests  were  conducted  to  determine  the 
feasibility  of  identifying  operator 
performance  measurement  points  within  the 
system.  Both  an  Operational  Sequence 
Diagram  (OSD)  and  a  Mission  Time  Line  (MTL) 
were  developed  for  the  Tactical 
Coordination  station  within  the  context  of 
a  standard  Evaluation  Mission. 

The  appropriate  scales  of  measurement  for 
various  task  types  are  as  follows: 

(1)  Absolute  deviation  from  standard 
percent  of  time  out  of  design  limits 
(continuous  control  tasks). 

(2)  Percent  of  incorrect  responses 
(discrete  control  tasks). 

(3)  Percent  settings  not  on  design  setting 
or  percent  outside  design  limits 
(pointer/symbol  positioning  tasks). 

(4)  Percent  of  decisions  agreeing  with 
judges'  established  decisions 
(technical  decision  tasks). 
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4.5  The  Mission  Time  Line  (MTL)  was  used  to 

test  the  feasibility  of  obtaining  estimates 
of  Tactical  Coordinates  Workload  in  the 
early  stages  of  work  with  the  P-3C.  The 
feasibility  of  the  method  is  demonstrated 
in  this  report,  but  the  procedures  utilized 
are  not  since  the  estimate  of  workload  from 
the  MTL  is  more  properly  a  part  of  the 
system  development  phase. 

5.4  Recommendations  are  made  with  regard  to  the 
need  for  assigning  personnel  with  authority 
and  means  for  accomplishing  human  factors 
requirements. 

(1)  Detailed  listing  of  operator  tasks  and 
OSD's  should  be  developed  and  kept 
current  throughout  the  development  and 
tests  of  the  systems.  Manual  perform¬ 
ances  and  tactical  decision  points 
should  be  identified  early  along  with 
scales  of  measurement  and  performance 
criteria. 

(2)  The  evaluator  must  be  oriented  toward 
obtaining  reliable  measures  under  test 
conditions.  The  evaluator  should 
strive  towards  the  most  representative 
conditions  possible  while  maintaining 
good  test  procedure. 

(3)  Reliable  performance  data  must  be 
obtained  by  which  the  criterion 
requirement  for  successful  performance 
can  be  compared  in  order  to  reach  an 
evaluative  judgement  as  to  the 
adequacy  of  design. 

(4)  The  determination  of  whether  an 
operator  can  successfully  accomplish  a 
sequence  of  task3  in  the  time 
available  in  the  actual  mission  must 
be  made  under  conditions  as  nearly 
representative  of  the  operational 
minimum  as  possible. 
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(5)  It  is  important  that  the  “evaluator 
set-up  observational  and  report 
procedures  to  determine  the  degree  to 
which  deviations  from  design 
procedures  occur  through  use  of 
earlier  learned  habits. 
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Detection — When  the  observer  becomes 
aware  of  something  in  the  field  of 
view. 

Orientation — Enough  of  the  target's 
outline  is  seen  to  distinguish  between 
its  longest  and  shortest  dimensions. 


1.3  The  paper  describes  a  particular  visual 
target  acquisition  model  originally 
developed  for  a  particular  air-to-ground 
application,  but  subsequently  extended  to 
other  air-to-ground  and  eir-to-air 
applications.  The  basic  building  block  of 
this  model  (and  all  other  visual 
acquisition  models)  is  the  capability  of 
the  human  eye.  Separate  modifying  branches 
are  added  to  this  basic  block  to  increase 
the  model's  versatility. 

3.1  The  model  treats  acquisition  data  as  a 
function  of  slant  range  to  target.  Four 
types  of  sequential  acquisition  events  are 
included  in  the  model: 

(1) 
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(3)  Recognition — The  outline  is  seen  with 
sufficient  clarity  to  establish  the 
general  classification  of  the  target, 
e.g.,  tank,  truck,  parked  aircraft, 
etc. 
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5.1  Test  Execution 

5.2  Data  Analysis 

5.3  Findings  Interpretation 
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(4)  Identification — Sufficient  detail  to 
establish  the  particular  kind  of  tank, 
truck,  or  whatever. 

Verification  of  this  model  using  actual 
test  data  have  been  very  good. 
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5.  Application  Components 
of  the  Process 

5.1  Test  Execution 

5.2  Data  Analysis 

5.3  Findings  Interpretation 
5.0  Conclusions  and 

Recommendations 


The  purpose  of  this  study  was  the 
development  of  the  general  procedures 
necessary  to  obtain  human  performance  data 
which  will  satisfy  a  prior  model  that 
includes  human  performance  data  in  models 
of  infantry  weapon  system  reliability. 

The  following  procedures  were  followed  in 
order  to  implement  the  study:  (a) 
development  of  a  taxonomy  for  weapon-system 
tasks  performed  by  the  operator;  (b) 
selection  of  an  infantry  weapon  system  for 
study;  (c)  analysis  of  maintanence  tasks  to 
insure  said  tasks  were  consistent  with 
taxonomic  terms;  (d)  development  of 
categories  for  human  performance  errors 
that  relate  to  hardware  failures;  (e) 
propose  a  study  to  collect  objective  data 
on  human  performance. 

The  following  methods  were  used  to 
implement  the  study. 

(1)  A  behavioral  taxonomy  was  developed 
which  divided  behavior  into  process, 
activities,  and  specific  behavior. 
Process  referred  to  groups  of  behavior 
associated  with  perception,  judgment, 
communication  and  muscular  activities; 
the  activities  column  classified  each 
of  these  processes;  the  "specific 
behavior"  column  showed  the 
system-oriented  behavior  associated 
with  each  category  of  each  process. 
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(2)  The  task  taxonomy  was  developed  from 
the  behavior  taxonomy.  Hardware 
elements  that  comprise  most  gun 
systems  were  identified;  row  headings 
representing  the  hardware  elements 
were  matched  with  column  headings 
representing  the  terms  in  the  behavior 
taxonomy;  the  cell  entries  represent 
performance  criteria. 

(3)  Human  performance  requirements  and 
human  error  categories  were  developed 
and  correlated  such  that  the  critical¬ 
ity  of  human  performance  error  was 
established  in  terms  of  its  probable 
effect  on  the  weapon. 

5.4  It  was  concluded  that  human  performance 
data  can  be  incorporated  into  the  models 
currently  available  that  seek  to  input 
human  engineering  factors  into  system 
reliability  models. 
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The  overall  purpose  of  this  study  was  to 
provide  a  method  of  using  cost  as  a  basic 
design  parameter  in  identifying  and 
defining  more  economical  space  transporta¬ 
tion  systems.  This  volume  reports  on  Task 
6  which  sought  to  determine  economically 
optimum  design  and  operational 
philosophies. 

Using  alternatives  defined  in  previous 
tasks,  systems  for  resupply  of  an  orbiting 
space  station  were  synthesized.  These 
systems  included  both  ballistic  and  lifting 
entry  vehicles  with  reuse  concepts. 

These  systems  were  analyzed  using  the 
Economic  Optimization  Model  to  determine 
the  effect  of  various  system  parameters  on 
total  program  cost  for  a  fixed  program 
size. 

Cargo  weight  per  launch  and  certain 
operational  modes  were  optimized. 
Sensitivity  of  program  cost  to  other 
operational  modes,  crew  size,  cargo 
density,  number  of  launches,  subsystem 
elements/return  time/orbit  inclination,  and 
number  of  landing  sites  were  also 
evaluated. 
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5.3  Findings  Interpretation 
5.4)  Conclusions  and 
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6.  Further  Research  Areas 
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This  report  presents  a  model  for  relating 
various  operating  and  design  parameters  to 
cost  and  by  using  search  and  optimization 
techniques,  finds  the  least  cost  system  for 
the  specified  parameters.  This  model  was 
developed  for  two  types  of  spacecraft  - 
ballistic  and  those  with  bodies  having  low 
lift-drag  ratios.  There  are  three  major 
modules  to  this  model.  The  first  is  the 
main  module  which  is  the  executive  control 
logic,  the  second  is  the  size  module  which 
translates  performance  and  operational 
requirements  into  a  vehicle  description  and 
weight  statement.  The  third  module  is  the 
cost  module  which  develops  the  total 
program  costs  from  the  data  supplied  by  the 
other  two  modules. 
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System  research  has  four  stages  or  purposes 
with  three  kinds  of  research  functions. 

The  four  stages  are:  delineation  of  system 
requirements,  delineation  of  design 
consequences  of  requirements,  system 
development  and  integration,  and  system 
evaluation.  The  three  types  or  research 
functions  are:  development  of  models, 
collection  of  research  information,  and 
synthesis  of  information. 

Two  general  types  of  models  were 
distinguished:  design  and  research  models. 
Successive  cycles  of  development  of  models, 
and  collection  of  information  and  synthesis 
of  information  serve  to  reduce  uncertainty 
in  the  formulation  of  research  problems  and 
to  delimit  a  reduced  matrix  of  potential 
design  solutions  from  which  an  optimal 
design  is  to  be  selected. 

The  information  gathering  and  synthesis 
function  includes:  determination  of 
relevant  variables,  determination  of  the 
range  of  values  of  these  variables,  and  the 
determination  of  the  interaction  of  the 
variables.  The  information  gathering  and 
synthesis  methods  differ  to  the  extent  to 
which  they  permit,  require  or  prohibit  the 
occurrence  of  values  of  relevant  variables; 
and,  thus,  the  amount  of  information  which 
potentially  can  be  obtained  from  them.  The 
efficiency  of  a  given  system  research 
method  can  be  assessed  in  terms  of  the 
total  amount  of  accountable  information 
about  the  system  research  problem  which  its 
application  can  provide  for  a  given  level 
of  research  effort. 
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A  project  officer's  primary  responsibility 
is  to  determine  whether  or  not  the  system 
undergoing  an  operational  test  can 
accomplish  its  assigned  mission  under 
realistic  fleet  conditions.  An  important 
aspect  of  system  performance  to  be 
considered  is  the  field  of  human  factors, 
that  is,  human  performance  in  relation  to 
system  performance. 

An  Operational  Evaluation  (OPEVAL)  is  the 
test  and  analysis  of  a  weapon  system, 
support  system,  component,  or  equipment 
conducted  by  the  Operational  Test  and 
Evaluation  Force,  under  service  operation 
conditions  insofar  as  practicable,  to 
determine  the  ability  of  a  system, 
component,  or  equipment  to  meet  specified 
operational  performance  requirements  and/or 
to  establish  suitability  for  service  use. 
When  appropriate,  an  operational  evaluation 
may  be  ordered  solely  for  the  development 
of  basic  tactical  doctrine,  training 
procedures,  and  requirements  for  training 
aids  and/or  countermeasures. 
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Application  Components 

(2) 

of  the  Process 

5.1  Test  Execution 

5.2  Data  Analysis 

(3) 

5.3  Findings  Interpretation 

5.**  Conclusions  and 

(K) 

Recommendations 


6.  Further  Research  Areas 
6.1  Measurement  System 
Limitations 
Research  Potentials/ 
Priorities 

6.3  Research  Planning 


OPEVAL  has  five  major  phases: 
Preparation  and  initial  planning. 
Devising  and  writing  the  test  plan. 
Conducting  the  test. 

Evaluation  of  data  from  the  test. 
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(5)  Derivation  of  conclusions  and 
preparation  of  the  final  report. 


2.4  The  '‘ollowing  were  identified  as  OPEVAL 

considerations  in  human  factors  testing: 

{ 1 )  While  human  factors  are  important, 
they  are  only  one  of  a  number  of 
important  factors  that  need  to  be 
considered  by  the  project  officer. 

(2)  It  is  unreasonable  to  ask  that  the 
project  officer  become  a  skilled  human 
factors  specialist.  He  cannot  trace 
all  of  the  implications  of  what  he 
observes,  nor  should  he  be  expected  to 
comprehensively  examine  exactly  how 
some  of  the  important  human  effects  he 
notes  can  be  brought  under  control. 

(3)  Most  test  environments  do  not  have  a 
large  enough  sample  of  men  for  the 
project  officer  to  get  a  clear 
indication  of  how  all  of  the  human 
factors  problems  arise.  He  cannot 
duplicate  the  fleet  personnel  problem 
on  a  single  ship. 

(4)  Many  human  factors  tests  are  extremely 
time-consuming  and  expensive  to  run 
because  all  of  the  situational 
conditions  must  be  repeated  exactly, 
with  different  men  working  within  the 
system. 

(5)  Practical  aborts  can  be  expected  which 
will  complicate  and  sometimes  negate 
the  project  officer's  attempts  to 
complete  a  test  of  anything,  including 
human  factors. 

(6)  The  most  efficient  use  of  the  project 
officer's  time  would  be  to  concentrate 
on  the  most  important  human  factors 
effects  and  to  gather  data  on  these 
effects.  Additional  associated 
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problems  must,  by  necessity,  remain 
the  province  of  the  human  factors 
specialist  who  can  assemble  large 
amounts  of  data,  carefully  study  the 
personnel  situation  in  general,  and 
draw  needed  conclusions. 

4.3  The  instruments  used  to  record  human 

performance  data  included  an  operations 
events  recorder  (the  model  in  this  study 
was  a  20  pen  Esterline  Angus),  recycling 
timers,  tape  recorders,  counters  (both 
manual  and  automatic),  and  function 
recorders. 


McRuer,  D.,  Jex,  H.R.,  Clement,  W.F.  &  Graham,  D.  A  systems  analysis  theory 
for  displays  In  manual  control  (STI  Tech  Rep.  163-1).  Hawthorne,  CA:  Systems 
Technology,  Inc.,  June  1968.  (AD-675  983). 
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1.  State  of  the  Art  Review  1.3 

of  the  Process 

1 . 1  General  System 
Measurements 

1.2  System  Taxonomy 
Model  (STM) 

1.3  Overall  Conceptual 
Process  Model  (CPM) 

2.  Contextual  Components 
of  the  Process 

2.1  System  Definition 

2.2  Mission  Definition 

2- 3  Environment  Definition 

2.4  General  Constraints 

2.5  Performance 
Requirements,  Ultimate 

2.6  Performance 
Criteria,  Ultimate 

3.  Analytic  Components 
of  the  Process 

3.1  Practical  Measurable 
Attributes 

3.2  Practical  Attribute 
Measures 

3- 3  Performance 

Requirements,  Specific 

3.4  Performance 
Criteria,  Specific 

3.5  Measurement  Procedures 

4.  Planning  Components 
of  the  Process 

4.1  Analytic  Methods 

4.2  Parameter  Determinations 

4.3  Apparatus  for  Testing 

4.4  Personnel  for  Testing 

4.5  Test  Plans 

5.  Application  Components 
of  the  Process 

5.1  Test  Execution 

5.2  Data  Analysis 

5.3  Findings  Interpretation 

5.4  Conclusions  and 
Recommendations 

6.  Further  Research  Areas 

6.1  Measurement  System 
Limitations 

6.2  Research  Potentials/ 

Priorities 

6.3  Research  Planning 


This  report  presents  the  basic  structure  of 
a  comprehensive  theory  for  the  development 
of  control  displays  for  pilots  of  manually 
controlled  vehicles.  This  theoretical 
framework  provides  a  paradigm  for  control 
display  development,  a  rational  basis  for 
experimental  programs,  a  theoretical 
foundation  for  analyzing  the  comparative 
merits  or  problems  of  new  operational 
display  systems,  and  for  the  generation  of 
new  display  concepts. 

The  theory  combines  several  manual  control 
developments  in  its  structure: 

(1)  The  vehicle  dynamics,  environmental 
disturbances,  command  structure,  and 
mission  criteria  derived  in  terms  of 
meaningful  servo  analysis  parameters. 

(2)  The  "best"  or  "alternative  best" 
feedbacks  for  the  pilot  are  derived 
using  the  "multiloop  feedback 
selection  hypothesis,"  which  includes 
the  human  operator’s  describing 
functions,  remnant,  and  subjective 
preferences. 

(3)  Quantitative  evaluation  of  the  system 
performance  measures,  information 
bandwidths,  and  stability  margins  made 
by  systems  analysis  techniques. 

(*0  The  required  display  resolution, 

scaling,  scanning  pattern  and  rates, 
and  workload  margins  are  derived, 
based  on  pilot  monitoring  and  scanning 
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models  in  terms  of  meaningingful 
sampling  analysis  parameters. 

(5)  Preferential  display  arrangements  are 
predicted  from  scanning  pattern  and 
workload  measures. 

(6)  Progression  and  regression  of  the 
level  of  pilot  behavior  (e.g. ,  during 
training,  transfer,  stress,  or 
equipment  failure)  are  treated  by  the 
successive  organization  of  perception 
theory  of  manual  control  skill 
development. 

The  theory  can  be  utilized  for  the 

following: 

(1)  Exposure  of  potential  problem  areas 
and  directions  for  improvement  at  an 
early  enough  stage  to  minimize  detail 
design  risks  and  costs. 

(2)  Prediction  of  the  best  display 
scaling,  filtering,  and  equalization 
parameters  (e.g.,  "quickening"). 

(3)  Analysis  of  display/pilot/vehicle 
system  instability  under  instrument 
flight  conditions  (e.g.,  flight  path 
oscillations  under  Instrument  Landing 
System  (ILS)  guidance). 

(*0  Selection  of  optimum  feedbacks  and 
their  gains  for  integrated  displays. 

(5)  Specification  of  display  instrument 
dynamic  range,  bandwidth,  and 
tolerable  dynamic  lags. 

(6)  Estimation  of  functional  limitations 
on  existing  instruments  as  applied  to 
new  missions,  vehicles,  and  tasks. 
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(7)  Optimization  of  the  location  of 
related  instruments  for  specific 
mission  phases. 

(8)  Rapid  post-mortem  investigation  of 
anomalies  in  measured  display  system 
performance. 

(9)  Analytical  evolution  of  relevant 
parameters  and  their  interactions,  as 
a  guide  to  the  experimental  design  of 
display  simulations  and  flight  tests. 

(10)  Correlation  and  unification  of  the 
results  of  numerous  ad  hoc  display 
tests  performed  in  the  past. 

(11)  Exposure  and  prediction  of  solutions 
to  new  problem  areas  in  integrated 
displays,  terrain-following  displays, 
etc. 

(12)  Evolution  of  display  simulator 
functional  requirements  (e.g., 
instrument  servo  lags,  cathode  ray 
tube  capabilities) . 

(13)  Interpretation  of  experimental 
findings  in  an  analytical  manner  to 
permit  their  extrapolation  to  actual 
flight  and  future  problems. 

(14)  Establishment  of  a  rational  basis  for 
cardinal  elements  on  contact  analog 
displays. 

(15)  Evolution  of  displays  to  resist 
disorientation  and  to  achieve  optimum 
head-up  display  arrangements. 

(16)  Synthesis  of  improved  blind-landing 
and  terrain-following  displays. 

(17)  Specification  of  the  best  training  and 
utilization  procedures  to  enhance 
learning  and  skill  transfer. 

An  example  is  presented  of  the  application 

of  the  theory  to  the  marjally  controlled 

ILS  approach  of  a  large  Jet  aircraft. 
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The  pragmatic  approach  to  predicting 
operational  performance  has  the  following 
characteristics:  (Da  conscious  attempt 
to  avoid  mathematical  models  and 
theoretical  processes  and,  instead,  to 
extrapolate  predictive  indices  directly 
from  empirical  data;  (2)  an  emphasis  on 
data,  not  theory;  (3)  a  focus  on  certain 
parameters  that  are  assumed  to  be  important 
to  operator  performance.  Logically  and 
heuristically,  it  can  be  assumed  that  a 
restricted  number  of  parameters  account  for 
the  greatest  part  of  the  operator's 
performance;  if  human  performance  were 
affected  equally  by  all  possible  factors, 
it  would  be  infinitely  variable  and 
unpredictable. 

The  important  parameters  tend  to  be  task 
oriented  or  at  least  related  to  operational 
system  requirements.  The  parameters 
selected  for  predicting  operational 
performance  obviously  define  what  data  are 
needed . 


1. 
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Planning  Components 
of  the  Process 

1.1  Analytic  Methods 

1.2  Parameter  Determinations 
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1.5  Test  Plans 

Application  Components 
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5.1  Test  Execution 
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5.3  Findings  Interpretation 

5.1  Conclusions  and 
Recommendations 

Further  Research  Areas 

6.1  Measurement  System 
Limitations 

6.2  Research  Potentials/ 
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6.3  Research  Planning 


In  predicting  performance,  it  is  also 
necessary  to  account  for  the  fact  that  more 
than  one  task  may  be  performed  concurrently 
by  the  same  operator.  Each  of  two 
concurrent  tasks  has  its  own  important 
parameters,  for  predicting  the  task's 
individual  performance;  the  concurrency  of 
the  two  tasks  is  another  important 
parameter  for  predicting  the  total 
performance. 

The  parameters  considered  to  be  important, 
in  general,  are  the  following: 
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(1)  The  number,  organization,,  and 
utilization  sequence  of  controls  and 
displays  involved  in  the  task. 

(2)  The  display  exposure  time. 

(3)  Display  visibility. 

(4)  The  nature  of  the  stimulus  displayed 
(i.e.,  structured  or  unstructured). 

(5)  The  number  of  visual  stimuli. 

(6)  The  operator  function  (i.e.,  the  type 
of  response  required  of  the  operator: 
e.g.  ,  discrete  control  response, 
continuous  control  response, 

simple  monitoring,  detection, 
discrimination,  tracking,  stimulus 
identification,  information 
extraction,  decision-making,  etc.) 

(7)  Stimulus  movement. 

(8)  The  characteristics  of  the 
control-display  coordination. 

(9)  The  amount  of  information  the  operator 
must  handle. 

(10)  The  feedback  (type,  amount)  provided 
to  the  operator  following  his  actions. 

Other  important  parameters  that  relate  to 

task  requirements  include: 

(1)  The  performance  accuracy  required  of 
the  operator. 

(2)  The  operator  loading  (pacing,  or  time 
constraints) . 
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4.2  Theoretically,  data  on  these  parameters  can 
be  obtained  under  either  experimental 
conditions  or  the  actual  operational 
environment.  The  author  doubts  that 
experimental  work  will  be  able  to  supply 
the  necessary  information,  given  the  poor 
record  of  human  factors  research  in 
supplying  relevant  data  stores,  and  given 
the  focus  of  most  research  on  operationally 
meaningless,  artificial,  unrealistic  tasks. 

The  difficulty  of  data  collection  in  the 
operational  environment  is  one  of  setting 
up  conditions  which  isolate  the  parameters 
of  interest:  the  parameters  usually  exist 
only  in  interaction  under  operational 
conditions.  The  solution  is  to  identify 
operational  conditions  which  display 
combinations  of  parameters  of  interest.  By 
locating  and  measuring  different  parameter 
combinations,  comparing  results,  and 
ascribing  differences  to  variations  in  the 
parameters  between  the  two  combinations, 
the  individual  parameter  effects  can  be 
isolated. 


Meister,  D.  Assumptions  underlying  test  and  evaluation  strategies.  San 
Diego,  CA:  Navy  Personnel  Research  and  Development  Center.  (Speech, 
undated) . 
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2.  Contextual  Components 
of  the  Process 
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2.5  Performance 
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2.6  Performance 
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3.1  Practical  Measurable 
Attributes 
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0.0  Personnel  for  Testing 
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of  the  Process 

5.1  Test  Execution  2.1 

5.2  Data  Analysis 
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Five  assumptions  are  discussed  at  length. 
They  are  summarized  below: 

(1)  Human  performance  can  be  measured  in  a 
system  work  context. 

(2)  Since  human  performance  occurs  as  part 
of  system  operations,  it  must  be 
measured  in  a  system-related  way. 

(3)  Maximum  simulation  fidelity  produces 
more  valid  tests  and  evaluations  than 
lesser  amounts  of  simulation  fidelity 
to  the  operational  system  and  environ¬ 
ment  in  which  the  system  being  tested 
will  perform. 

(4)  Test  and  Evaluation  (TE)  is  important 
only  during  a  specific  period  of 
system  development. 

(5)  It  is  possible  to  conduct  TE  without 
specifying  in  advance  human  perform¬ 
ance  criterion  values.  The  primary 
question  asked  in  any  TE  situation  is: 
Is  personnel  performance  satisfying 
overall  system  requirements? 

It  should  be  noted  that  assumptions  3  to  5 
are  either  elaborated  on  and/or  refuted  by 
the  author. 

Test  and  evaluation,  as  the  author 
described  it,  is  termed  OST  (Operational 
System  Testing)  in  which  the  new  system  is 
a  prototype  placed  in  a  functional  setting 
that  is  or  resembles  the  environment  it 
will  ultimately  be  exposed  to. 
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3.3  Performance 
Requirements,  Specific 

3.4  Performance 
Criteria,  Specific 

3.5  Measurement  Procedures 

4.  Planning  Components 
of  the  Process 

4.1  Analytic  Methods 

4.2  Parameter  Determinations 
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The  author  points  out  that  any  measurements 
of  system  reliability  or  system 
effectiveness  which  do  not  include  indices 
of  human  performance  must  necessarily 
produce  an  erroneous  estimate  of  that 
system's  reliability  or  effectiveness.  The 
goal  of  the  human  reliability  methodology 
is  to  quantify  human  performance  in  a 
man-machine  context. 

The  author  notes  that  these  procedures 
derive  by  analogy  from  an  equipment 
reliability  orientation;  the  human 
component  is  viewed  as  a  "black  box",  and 
human  performance  is  conceptualized  in 
terms  of  a  single  probabilistic  measure, 
i.e.,  task  completion  or  task  failure. 

This  concept  also  implicitly  assumes  that 
each  stimulus  condition  presented  to  the 
subject  increases  the  potential  for  error 
and  reduces  the  probability  of  successful 
task  completion. 

Another  assumption  of  the  model  is  that  the 
individual  probabilities  of  performance  for 
the  behavioral  units  summate  mutiplica- 
tively;  however,  when  combining  task 
probabilities  to  derive  higher  function 
probabilities,  it  is  unreasonable  to  assume 
independence  of  those  tasks  implicit  in 
multiplicative  summation. 

The  model  also  makes  the  overly  simplistic 
assumption  that  error  is  equivalent  to 
failure  to  complete  the  task;  however,  it 
is  perfectly  obvious  that  a  task  might  be 
completed  successfully  even  if  several 
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errors  are  made  during  its  pecformanee 
because  the  human  operator  often  can 
correct  his  errors. 

The  model  also  has  the  inadequacy  of 
dealing  only  with  individual  operator 
behavior;  it  forms  predictions  of  group 
performance  through  the  straight 
miltiplieative  process,  and  this  is 
manifestly  inappropriate. 

The  model  also  assumes  no  change  in  the 
behavioral  unit  probabilities  as  a  function 
of  repeated  performance,  and  takes  no 
account  of  individual  variations  in  skill. 

3.5  The  major  steps  in  the  application  of  the 
human  reliability  model  are  as  follows: 

(1)  Analyze  system  operations  into 
discrete  units  of  behavior  to  which 
predictive  data  can  be  applied. 

(2)  Determine  the  parameters  affecting 
each  task  behavioral  unit. 

(3)  Assign  probability  values  based  on 
historical  data  to  each  behavioral 
unit. 

(4)  Combine  the  individual  unit  probabili¬ 
ties  together  to  form  task 
probabilities,  function  probabilities, 
etc. 

5.4  In  the  summary,  it  is  noted  that  many  of 
the  assumptions  in  this  model  are 
consciously  incorrect,  and  therefore  it  is 
questioned  whether  this  knowingly 
inadequate  model  still  has  value.  The 
answer  is  yes,  and  that  it  has  two  types  of 
values:  First,  it  stimulates  conceptual 
activity  by  the  investigators  and  raises 
scientifically  important  questions  which 
might  not  have  been  raised  without  the 
model.  Second,  it  has  heuristic  value,  in 
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that  is  reduces  the  amount  of  uncertainty 
that  we  have. 


The  paper  discusses  the  human  reliability 
model  primarily  as  a  means  of  illustrating 
certain  characteristics  of  behavioral 
models  in  general  and  certain  characteris¬ 
tics  of  model-makers  themselves.  The 
author's  personal  view  is  that  a  model  is 
effective  to  the  extent  that  it  helps 
either  to  gather  data  and/or  to  explain 
those  data.  As  a  corollary,  he  states  that 
any  behavioral  model  which  is  not  concerned 
with  real-world  data  (as  opposed  to  labora¬ 
tory  data)  is  not  useful.  However,  he 
observes  that  behavioral  models  characteris¬ 
tically  have  employed  laboratory  data  and 
ignored  or  been  unable  to  handle  natural 
event  data.  Models  thus  are  painfully  data 
limited  -  -  their  characteristics  and  their 
basic  assumptions  often  are  determined  by 
the  limitations  of  the  data  input  to  them. 
These  data  limitations  often  result  from 
the  way  in  which  we  scientists  have  been 
taught  to  perform  our  studies  and  by  the 
biases  we  knowingly  or  unknowingly  insert 
into  these  studies. 

The  human  reliability  model  is  cited  as  a 
case  in  point.  The  author  asserts  that 
this  model's  assumptions  derive  from  the 
unsystematic  manner  in  which  the  model's 
input  data  were  secured,  and  he  points  out 
that  at  least  in  part,  these  assumptions 
demonstrably  are  not  in  accord  with 
empirical  reality. 


•  *  «  ■  ^  • 
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1.1  The  concept  of  " system"  is  the  foundation 
of  human  factors  studies  which  seek  to 
measure  factors  that  affect  personnel 
performance  in  manned  systems.  In 
measuring  any  system,  one  must  keep  in  mind 
certain  universal  properties  of  all 
systems:  they  are  organized 
hierarchically,  have  a  purpose,  are 
interdependent,  and  contain  their  own 
standards  for  measurement,  evaluation,  and 
feedback  since  a  purposive  system 
establishes,  ipso  facto,  its  own  standards. 

1.3  Looking  at  operational  manned  systems  in 
general,  one  can  list  several  aspects  of 
the  measurement  process  that  are  common  to 
any  analysis: 

(1)  Assessment  of  the  impact  of  system 
parameters  on  personnel  performance. 


3-3 

Performance 

Requirements,  Specific 

(2) 

3.* 

Performance 

Criteria,  Specific 

3-5 

Measurement  Procedures 

(3) 

Planning  Components 

of  the  Process 

4.1 

Analytic  Methods 

4.2 

Parameter  Determinations 

4.3 

Apparatus  for  Testing 

(*0 

4.14 

Personnel  for  Testing 

4.5 

Test  Plans 
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stimulus  to  end-point). 

Replication  (validation)  of  the 
research  study  under  identical  or 
simulated  conditions. 
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All  system-relevant  factors  must  be 
included  in  any  measurement  situation  to 
ensure  that  those  variables  chosen 
represent  the  operational  system.  There 
are  two  factors  involved:  (1)  all 
variables  that  would  be  expected  to  affect 
system  output  in  the  operational  environ¬ 
ment  must  be  included;  (2)  all  interactions 
in  the  system  representation  must  also  be 
included. 

The  type  of  measurement  system  focused  on 
here  was  called  PSM  or  Personnel  Subsystem 
Measurement:  measurement  of  personnel 
performing  a  task  or  job  within  the  actual 
work,  or  system,  environment. 

PSM's  purpose  was  to  determine  the 
feasibility  of  a  research  approach;  choose 
an  optimal  alternative;  determine 
performance  capability;  resolve  personnel 
subsystem  dysfunctions;  contribute  needed 
research. 

A  comprehensive  use  of  PSM  has  the 
potential  to  answer  the  following  questions 
pertaining  to  four  separate  stages:  system 
development,  training  program  development, 
system  operation,  and  system  maintenance. 

(1)  System  Development: 

a.  Do  personnel  have  the  capability 
to  perform  certain  tasks  at  a 
specified  level? 

b.  Is  there  an  optimally  efficient 
interface  between  system  design 
and  personnel? 

c.  Regarding  personnel  performance, 
two  or  more  system  configurations 
is  most  effective? 
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b.  How  efficient  is  that 
maintenance? 

3.1  In  using  measures  to  evaluate  system 
performance,  PSM's  approach  is  to  measure 
the  individual,  team,  subsystem,  and  system 
levels  where  appropriate.  PSM  emphasizes 
normative  data,  which  are  primarily 
descriptive. 

Two  types  of  normative  data  were  used  in 
PSM  evaluation: 

(1)  Data  describing  systems  and  their 
inter elationships. 

(2)  Data  describing  personnel  task 
performance. 

4.1  Data  are  analyzed  by  correlational  analysis 
in  PSM  under  operational  conditions;  in  a 
laboratory  experiment,  testing  the 
significance  of  differences  is  the  most 
common  technique. 

4.2  The  parameters  of  a  given  PSM  evaluation 
involved  system  and  personnel 
specification.  With  regard  to  personnel 
parameters,  the  following  must  be  defined: 
characteristics  of  equipment,  job, 
individual  aptitude,  skill,  experience,  and 
motivation. 
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This  handbook  has  been  developed  to  enable 
Marine  Corps  test  personnel  to  evaluate  a 
new  system  for  its  human  factors  adequacy. 
The  term  "human  factors"  refers  to  the 
entire  complex  of  elements  that  affect 
personnel  performance. 

This  handbook  provides  information  on  the 
Personnel  Performance  Test  Plan,  including 
its  purpose,  a  step-by-step  description  of 
its  various  sections  and  requirements  for 
completing  those  sections,  a  model  for 
illustrating  the  material  to  be  included  in 
such  a  plan  and  a  procedure  for  developing 
quantitative  personnel  performance 
criteria. 

The  next  section  covers  the  selection  and 
development  of  measures  and  measurement 
methods  followed  by  a  human  engineering 
checklist  of  procedures.  The  handbook  also 
describe  "Self  Report  Rating  Scales"  and 
interview  questions.  Test  procedures  are 
covered  and  there  is  also  an  introduction 
to  statistical  methodology.  The  last 
sections  contain  a  "Personnel  Performance 
Test  Planner’s  Checklist"  and  a  Personnel 
Performance  Test  Report.  The  final  section 
gives  references  which  might  be  useful  to 
the  user  of  this  handbook. 
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1.1  The  practice  of  human  factors  in  the 

relationship  between  the  capabilities  and 
limitations  of  men  and  the  characteristics 
of  machines  involves  the  following: 

(1)  Performance  of  a  functional  analysis 
of  system  requirements. 
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(2)  Development  of  personnel  selection  and 
training  criteria. 

(3)  Design  and  evaluation  of  manually 
operated  control  and  crew  personal 
equipment. 

(4)  Study  of  the  environmental  factors 
affecting  human  performance. 

(5)  Participation  in  system  performance 
tests. 

(6)  Surveillance  of  production  facilities 
and  applied  research. 

Whenever  there  are  design  alternatives, 
hunan  factors  evaluation  is  essential  to 
eliminate  the  systems'  susceptibility  to 
human  error. 

It  is  difficult  to  compare  the  man-machine 
behavior  of  one  system  with  that  of  another 
because  the  terms  of  that  behavior  are  not 
yet  operationally  definable. 
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A  crucial  characteristic  of  the  system  is 
its  purposiveness  (goal-directed  behavior). 
Goals  can  be  divided  into  two  groups: 
primary  (mission-oriented)  and  secondary 
(supporting).  Primary  goals  seek  to 
accomplish  the  system  mission  and  direct 
the  performance  of  all  mission-related 
system  activities.  Secondary  goals  seek  to 
maintain  the  integrity  of  the  system  until 
the  mission  has  been  accomplished. 

The  mission  described  the  man-machine 
activities  performed  to  accomplish  the 
primary  system  goals.  Unless  it  is  framed 
in  terms  of  mission  goals,  system  behavior 
becomes  extremely  difficult  to  explain  or 
understand  because  purpose  is  the  single 
factor  which  unifies  a  great  variety  of 
disparate  system  behavior. 
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A  problem  to  systems  designers  is  the 
allocation  of  functions  between  man  and 
machine.  This  paper  reports  an  attempt  to 
isolate  and  identify  factors  pertinent  to 
making  allocation  decisions.  In  an 
analysis  of  the  functions  and  missions  of 


several  automated  systems,  the  following 
factors  were  shown  to  be  highly  relevant 
allocation  decisions: 

(1) 

State 

of  the  art. 

(2) 

Cost 

of  automation. 

(3) 

Space 

and  weight  constraints. 

(4) 

Lead 

time. 

(5) 

Difficulty  to  program. 

(6) 

Man's 

role  in  automated  systems. 

The  designer  of  an  automated  system  needs  a 
clear-cut  conception  of  the  general  role  of 
man  in  such  systems.  In  effect,  the 
designer  needs  a  conception  of  what  man's 
role  should  be  before  he  can  decide  what  it 
will  be. 
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5.4  Conclusions  and 
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2.1  The  primary  purpose  of  this  document  is  the 
development  of  a  "surface  task  analysis" 
based  on  a  sample  of  sixteen  tactical 
maneuvers  found  in  tactical  flying  (seven 
air-to-ground  maneuvers  and  nine 
air-to-air) . 
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2.3  The  authors  chose  the  F-4E  aircraft  as  a 
representative  tactical  craft,  and  based 
their  analyses  on  pilot  performance  in  the 
F-4E. 

2.5  Ultimately,  the  use  of  a  surface  task 

analysis  will  allow  a  complete  description 
of  a  flying  task  or  maneuver,  step-by-step. 
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By  focusing  on  the  pilot's  aircraft  control 
behavior,  this  study  offers  a  data  system 
from  which  improved  flying  training 
concepts  and  methods  can  be  derived.  Using 
this  system  (surface  task  analysis), 
training  developers  can  determine  and 
substantiate  the  content  of  training 
programs.  Training  programs  can  also  be 
identified  and  alternative  solutions 
formulated  using  this  system. 

The  surface  analysis  technique,  by  defini¬ 
tion,  measures  the  following:  (a)  cues,  or 
inputs  which  the  pilot  received  from  his 
flying  environment  to  perform  a  certain 
task;  (b)  mental  action,  which  processed 
the  cues,  and  (c)  motor  actions,  or  outputs 
in  the  form  of  movements  of  the  aircraft 
flight  controls. 


6.  Further  Research  Areas 

6.1  Measurement  System 
Limitations 

6.2  Research  Potentials/ 
Priorities 

6.3  Research  Planning 


A-300 


ABSTRACT 


The  cues  were  divided  into  four  categories: 
visual,  aural,  control,  and  motion  cues. 
Visual  cues  included  outside  or 
environment-generated  cues,  such  as  the 
horizon  or  cloud  formation;  inside,  or 
aircraft-generated  cues,  includes 
information  obtained  from  flight 
instrumentation  or  radar.  Aural  cues 
include  such  items  as  communication, 
weapons  tones,  slipstream,  etc.  Control 
cues  involved  aircraft  flight  controls, 
which  were  either  (a)  dynamic  tactual, 
e.g.,  aileron,  stabilizer,  rudder, 
throttle,  or  (b)  discrete  tactual,  e.g., 
toggle  and  rotary  switches,  etc.  Finally, 
motion  cues  consisted  of  stimuli  which  were 
sensed  by  body  receptors  such  as  +  or  - 
G-force,  vibration,  etc. 

Rules  and  procedures  were  developed  for  the 
specific  application  of  the  cues  categories 
in  performing  a  surface  task  analysis. 
Generally  speaking,  each  surface  analysis 
must  identify  the  following:  (a)  the 
aircraft  type  involved;  (b)  the  maneuver(s) 
involved  and  weapons  delivery;  (c)  whether 
the  maneuver  environment  is  a  range-  or 
tactically-oriented  one;  (d)  the  flight 
path(s)  of  the  aircraft;  (e)  the  starting 
situation  of  the  aircraft  and  the  specified 
task  goal.  A  diagram  of  the  maneuver  is 
then  prepared,  and  for  each  element  of  the 
flight  sequence,  a  chart  is  filled  in  with 
information  on  cues,  mental  action,  and 
motor  action. 

The  data  base  for  the  entire  taxonomy 
classification  consisted  of  interviews  with 
pilots  regarding  sixteen  representative 
tasks,  both  air-to-ground  and  air-to-air. 
There  are,  therefore,  sixteen  separate 
surface  task  analyses.  Detailed  analyses, 
or  classifications  of  flight  maneuvers  were 
given  to  complete  Volume  I  of  the  study. 
Volume  II  details  the  rationale  and  methods 
used  to  formulate  a  taxonomic  structure  for 
tactical  flying  tasks. 


Meyer,  R.P.,  Laveson,  J.I.  4  Pape,  G.L.  Development  and  application  of  a 
task  taxonomy  for  tactical  flying  (AFHRL-TR-78-1*2.  Vol  2).  St.  Louis,  MO: 
Design  Plus,  September  1978.  (AD-061  388). 


Topics  Relevant  i  i 

to  System  Development  i Topic! 

and  Evaluation  Technology  1  No.  i _ ABSTRACT 

1.  St»te  of  the  Art  Review 
of  the  Process 

1.1  Genersl  System 
Measurements 

1.2  System  Taxonomy 
Model  (STM) 

1.3  Overall  Conceptual 
Process  Model  (CPM) 

2.  Contextual  Components 
of  the  Process 

2.1  System  Definition 

2.2  Mission  Definition 

2.3  Environment  Definition 

2.4  General  Constraints 

2.5  Performance 
Requirements,  Ultimate 

2.6  Performance 
Criteria,  Ultimate 

3.  Analytic  Components 
of  the  Process 

3.1  Practical  Measurable 
Attributes 

3.2  Practical  Attribute 
Measures 

3.3  Performance 
Requirements,  Specific 

3.4  Performance 
Criteria,  Specific 

3.5  Measurement  Procedures 

4.  Planning  Components 
of  the  Process 

4.1  Analytic  Methods 

4.2  Parameter  Determinations 

4.3  Apparatus  for  Testing 

4.4  Personnel  for  Testing 

4.5  Test  Plans 

5.  Application  Components 
of  the  Process 

5.1  Test  Execution 

5.2  Data  Analysis 

5.3  Findings  Interpretation 

5.4  Conclusions  and 
Recommendations 

6.  Purther  Research  Areas 

6.1  Measurement  System 
Limitations 

6.2  Research  Potentials/ 

Priorities 

6.3  Research  Planning 


1.3  A  taxonomy  of  tactical  flying  skills  was 
developed  as  a  user-oriented  skill-task 
analysis  system  for  practical  application 
in  solving  Tactical  Air  Command  (TAC) 
continuation  training  problems  and  for  a 
behavioral  data  base  for  skill  maintenance 
and  reacquisition  training  research  and 
development.  Sixteen  representative 
tactical  air-to-air  and  air-to-surface 
maneuvers  were  analyzed  and  classified 
within  the  system  with  provision  for  later 
expansion.  A  classification  system  was 
developed  to  accommodate  the  complexities 
of  tactical  flying.  A  data  system  was 
organized  with  sufficient  flexibility  to 
objectively  address  many  areas  of  tactical 
flying.  The  taxonomy  system  also  included 
methodology  for  addressing  on-going 
training  problems  and  requirements. 
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A  taxonomy  of  tactical  flying  skills  was 
developed  as  a  user  oriented  skill-task 
analyses  system  for  practical  application 
in  solving  Tactical  Air  Command  (TAC) 
continuation  training  problems  and  for  a 
behavioral  data  base  for  skill  maintenance 
and  reacquisition  training  research  and 
development.  Sixteen  representative 
tactical  air-to-air  and  air-to-surface 
maneuvers  were  analyzed  and  classified 
within  the  system  with  provisions  for  later 
expansion. 
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1.1  The  cost-effectiveness  of  human  factors 
(HF)  testing  in  any  material  development 
project  depends  in  large  part,  on  the  uses 
to  which  the  test  data  will  be  put. 

Whereas  the  human  engineer  contributes  data 
in  terms  of  human  performance  or  effects  of 
specific  variables,  the  systems  analyst 
wants  the  question  "So  what"  explained  in 
numbers.  AMSAA  (Army  Material  Systems 
Analysis  Activity)  has  stated  that  the 
"soldier  is  part  of  the  system"  and  human 
factors  data  should  be  analyzed  "not  as  a 
separate  additional  activity,  but  as  an 
integral  part  of  our  evaluation  of  each 
system".  The  HF  data  therefore,  need  to 
fit  into  five  categories  of  interest  to  the 
systems  analyst:  effectiveness,  reliabili¬ 
ty,  availability,  maintainability,  and 
training. 

3.1  The  normal  performance  measures  used  to 
determine  figures  for  the  above  five 
categories  were  time  and  error. 
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1.  State  of  the  Art  Review 
of  the  Process 

1 . 1  General  System 
Measurements 

1.2  System  Taxonomy 
Model  (STM) 

1.3  Overall  Conceptual 
Process  Model  (CPM) 

2.  Contextual  Components 
of  the  Process 

2.1  System  Definition 

2.2  Mission  Definition 

2.3  Environment  Definition 

2.4  General  Constraints 

2.5  Performance 
Requirements,  Ultimate 

2.6  Performance 
Criteria,  Ultimate 


1.2  The  author  examines  all  biological  and 

social  systems  and  divides  them  into  seven 
hierarchical  levels:  cells;  organs 
(composed  of  cells);  organisms  (independent 
life  forms);  groups  (families,  committees, 
etc.);  organizations  (communities, 
universities);  societies  or  nations;  and 
supranational  systems.  Each  of  these 
levels  carries  on  both  matter-energy  and 
information-processing  functions  for  a 
total  of  19  critical  subsystems. 

The  author  defines  13  distinct  concepts 
which  must  be  understood  in  analyzing  any 
living  system  at  any  level: 


3.  Analytic  Components 
of  the  Process 

3.1  Practical  Measurable 
Attributes 

3.2  Practical  Attribute 
Measures 

3-3  Performance 

Requirements,  Specific 
3 .  H  Performance 

Criteria,  Specific 
3.5  Measurement  Procedures 

4.  Planning  Components 
of  the  Process 

4.1  Analytic  Methods 

4.2  Parameter  Determinations 

4.3  Apparatus  for  Testing 

4.4  Personnel  for  Testing 

4.5  Test  Plans 


(1)  Space  and  Time:  all  concrete  systems, 
living  and  non-living,  exist  in 
physical  space;  this  latter  space  acts 
in  various  ways  to  partially  constrain 
and  determine  the  system  under 
consideration.  For  example,  residents 
of  a  neighborhood  who  are  physically 
closer  in  space  to  each  other  will 
interact  with  a  greater  rate  of 
frequency;  alternatively,  the  number 
of  nucleotide  bases  -  which  are 
configurations  in  space  -  that  a  DNA 
molecule  contains  will  determine  the 
amount  of  information  (bits)  that  it 
can  contain. 


5.  Application  Components 
of  the  Process 

5.1  Test  Execution 

5.2  Data  Analysis 

5.3  Findings  Interpretation 

5.4  Conclusions  and 
Recommendations 

6.  Further  Research  Areas 

6.1  Measurement  System 
limitations 

6.2  Research  Potentials/ 
Priorities 

6.3  Research  Planning 


Examples  of  conceptual  space  which  can 
be  found  in  all  living  and  nonliving 
systems  include:  (1)  Lev/in's  "life 
space,"  which  is  the  perceived  environ 
ment  that  immediately  surrounds  an 
individual;  and  (2)  Warner's  "social- 
space"  which  separates,  say,  an 
upper-lower  class  person  from  a 
lower-middle  class  person. 
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The  concept  of  time  has  been 
acceptably  quantified  through  everyday 
use  of  seconds,  minutes,  etc.,  to 
measure  durations.,  speeds, 
accelerations,  etc.  However,  the 
general  theory  of  relativity  makes  it 
clear  that  for  large  systems,  e.g.  , 
astronomical  systems,  time  cannot  be 
measured  on  any  absolute  scale. 

(2)  Matter  and  Energy:  Matter,  defined  as 
anything  which  has  mass  and  occupies 
physical  space  may  have  several  forms 
of  energy,  defined  in  physics  as  the 
ability  to  do  work:  (a)  Kinetic 
energy,  when  mass  is  moving  and 
creates  a  force  on  other  matter;  (b) 
potential  energy,  referring  to  a  mass' 
position  in  a  gravitational  field,  and 
(c)  rest  mass  energy,  which  results 
from  the  conversion  of  matter  into 
energy.  Matter  tjnd  energy  are 
equivalent  (E=MC  ;;  all  living  systems 
require  specific  types  of 
matter-energy. 

(3)  Information:  here  defined  as  "  .  .  . 
the  degrees  of  freedom  that  exist  in  a 
given  situation  to  choose  among 
signals,  symbols,  messages,  or 
patterns  to  be  transmitted."  The  term 
"marker"  is  used  to  denote  an 
observable  unit  or  change  of 
matter-energy  whose  patterns  indicate 
informational  symbols  (e.g.,  cuneiform 
writing,  punched  cards,  pulses  on  a 
telegraph  wire);  due  to  advances  in 
technology,  the  matter-energy  costs 
of  storing  and  transmitting  these 
markers  has  decreased,  while 
information-processing  efficiency  has 
increased  by  decreasing  the  mass  of 
these  markers:  cuneiform  tables 
carried  10“  bits  of  information  per 
gram;  paper  with  typed  messages  10^ 
bits  per  gram;  electronic  magnetic 
tape  storage  carries  10°  bits  per 
gram.  The  point  to  be  made  for 
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systems  analysis  is  that  there  are 
significant  matter-energy  constraints 
upon  information-processing  of  all 
living  systems  as  a  result  of  the 
composition  of  their  "markers." 

(4)  System:  the  most  general  definition 
is  "...  a  set  of  interacting  units 
with  relationships  among  them." 

Systems  may  be  open  (boundaries  may  be 
crossed  by  information  or 
matter-energy  transmissions),  closed, 
nonliving,  living,  totipotential  (can 
carry  out  all  critical  subsystem 
processes  necessary  for  life), 
partipotential,  fully  functioning,  or 
partially  functioning.  Each  type  of 
system  responds  to  varying  types  of 
measurement. 

(5)  Structure:  "...  the  arrangement  of 
its  subsystems  and  components  in 
three-dimensional  space  at  a  given 
moment  in  time." 

(6)  Process:  changes  in  the  matter-energy 
or  information  in  a  system  is  called  a 
process.  A  process  is  reversible  if 
it  remains  the  same  regardless  of  the 
time  dimension.  Process  includes  the 
function  of  a  system  and  its  history. 

(7)  Type:  refers  to  similar 
characteristics  among  a  number  of 
living  systems. 

(8)  Level:  level  refers  to  the  particular 
position  of  a  system  within  a 
hierarchy,  e.g.,  groups  are  composed 
of  organisms,  organizations  are 
composed  of  groups,  etc. 

(9)  Echelon:  this  refers  to  the  various 
levels  of  components  in  a  system  or 
subsystem.  Certain  decisions  are  made 
by  one  component  of  a  subsystem  and 
others  by  another,  the  two  components 
being  either  at  the  same  or  different 


■  *  •  N  '  *  • 

.'•VA'sV- 


- 


A-307 


Topics  Relevant  1 

to  System  Development  i  Topic 
and  Evaluation  Technology  i  Ho. 


ABSTRACT 


echelons  of  the  subsystem,. 

(10)  Suprasystem:  this  refers  to  the  next 
highe  •  system  in  which  the  system  in 
question  is  a  subcomponent  or  part. 

The  immediate  environment  of  a  system 
should  be  distinguished  from  the 
suprasystem:  the  latter  will  affect 
both  the  system's  environment  and  the 
system  itself. 

(11)  Subsystem  and  component:  subsystems 
are  distinct  structures  which  carry 
out  distinct  processes  within  a 
system.  The  distinct,  structural 
units  themselves  are  called  the 
components.  Within  the  level  of 
organizations,  the  concept  of  role 
would  be  a  component;  however,  not 
only  will  the  role  affect  the 
organization,  but  the  nature  of  the 
individual  who  fills  that  role  will 
have  an  impact  as  well. 

(12)  Transmissions  in  Concrete  Systems: 
Transmissions  may  be  of  matter, 
energy,  or  information,  and  can  be 
analytically  distinguished  in  terms  of 

(a)  inputs  across  system  boundaries, 

(b)  internal  processes  within  a 
system,  and  (c)  system  outputs.  The 
"template"  of  a  system  is  its  original 
genetic  or  information  input  that 
programs  the  system's  structure  and 
process. 

(13)  Steady  State:  all  living  systems 
maintain  steady  states  (homeostasis) 
among  their  variables,  keeping  a 
balance  not  only  with  intra-system 
variables,  but  with  their  environments 
and  suprasystems  as  well.  The 
"LeChaterlier  principle"  is 
particularly  apropos  to  the  steady 
state  concept,  and  states  that  a 
stable  system  under  stress  will  move 
in  that  direction  which  minimizes  the 
effect  of  stress  (a  compensatory  force 
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will  be  exerted  to  oppose  the  stress, 
usually  accompanied  by  changes  in 
related  system  variables  that  are  not 
directly  affect'd  by  the  stress). 
Cybernetics,  the  study  of  methods  of 
feedback  control,  is  an  important  part 
of  steady  state  theory. 
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1 .  State  of  the  Art  Review 
of  the  Process 

1 . 1  General  System 
Measurements 

1.2  System  Taxonomy 
Model  (STM) 

1.3  Overall  Conceptual 
Process  Model  (CPM) 


1 . 1  This  paper  addresses  the  problems  of 
assessing  worth  among  alternative 
transportation  systems.  It  assumes  that  a 
decision  context  has  been  specified  and 
that  a  fixed  set  of  discrete  alternatives 
has  been  produced. 


2.  Contextual  Components  2.5 

of  the  Process 

2.1  System  Definition 

2.2  Mission  Definition 

2.3  Environment  Definition 

2.4  General  Constraints 

2.5  Performance  2  6 

Requirements,  Ultimate 

2.6  Performance 
Criteria,  Ultimate 

3.  Analytic  Components 
of  the  Process 

3.1  Practical  Measurable  3-3 

Attributes 

3.2  Practical  Attribute 
Measures 

3-3  Performance 

Requirements,  Specific 

3.4  Performance 
Criteria,  Specific 

3.5  Measurement  Procedures 

4.  Planning  Components 

of  the  Process  o  j. 

4.1  Analytic  Methods  ^ * 

4.2  Parameter  Determinations 

4.3  Apparatus  for  Testing 

4.4  Personnel  for  Testing 

4.5  Test  Plans 

5.  Application  Components 
of  the  Process 

5.1  Test  Execution 

5.2  Data  Analysis 

5.3  Findings  Interpretation 

5.4  Conclusions  and 
Recommendations 


The  performance  requirements  are  that  the 
criteria  are  complete,  mutually  exclusive, 
of  major  significance  and  free  of  worth 
interdependence. 

The  first  step  of  the  assessment  procedure 
is  to  define  explicitly  what  is  desired  in 
the  way  of  performance  for  each 
alternative. 

Having  established  a  list  of  overall 
objectives,  the  second  step  is  to  generate 
a  hierarchial  structure  of  successively 
more  specific  performance  criteria.  This 
involves  breaking  down  or  subdividing 
higher  level  criteria  into  one  or  more 
lower-level  criteria  alleged  to  be  included 
within  the  meaning  thereof. 

The  third  step  is  to  select  a  physical 
performance  measure  for  each  lowest-level 
criterion.  Performance  measures  describe 
what  an  alternative  can  deliver,  while 
performance  criteria  state  what  the 
decision-maker  desires.  The  purpose  of 
selecting  performance  measures  is  to 
establish  concrete  connections  between 
desires  and  deliverable  performance  from 
real  alternatives. 


6.  Further  Research  Areas  3.5  In  the  fourth  Step  Specific  worth 

6.1  Measurement  System  relationships  are  mapped  out  between  each 

Limltitlons 

6.2  Research  Potentials/ 

Priorities 

6.3  Research  Planning 
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lowest-level  criterion  and  its  related 
performance  measure.  This  is  implemented 
by  defining  scoring  functions  which  assign 
a  unique  numerical  worth  score  to  every 
possible  value  of  a  performance  measure. 

The  worth  scores  are  then  combined  to 
arrive  at  an  overall  index  of  each 
alternative's  worth.  This  is  accomplished 
by  defining  a  weighting  function.  An 
additive  function  with  constant  trade-off 
weights  can  be  adopted  for  this  purpose. 

The  sixth  step  is  to  validate  both  the 
scoring  functions  and  the  weighting 
functions  against  whatever  alternatives 
have  been  produced.  This  means  computing 
an  overall  worth  index  for  each  alternative 
and  Judging  the  results  for  reasonableness. 
Results  generated  during  early  passes 
through  the  procedure  are  usually 
unreasonable  in  some  respect.  Selective 
revisions  by  use  of  the  following  might  be 
needed:  adding  criteria,  further  defining 
or  subdividing  existing  criteria, 
re-calibrating  scoring  functions  and 
re-adjusting  weights. 

The  final  process  is  to  assess  each 
alternative  by  trading  off  each's  overall 
worth  by  the  following  considerations: 
risk,  required  resource  expenditures, 
temporal  changes  in  objectives,  aspiration 
levels  and  tastes,  and  different  and 
possibly  conflicting  points  of  view  among 
diverse  interest  groups. 

An  experimental  test  of  the  procedure  was 
included  in  the  report. 
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Attempts  to  quantify  human  performance  in 
man-made  systems  are  generally  of  a  system 
reduction  form;  reductionists  means  that  a 
system  is  analyzed  into  modular  task 
elements.  Following  this  analysis,  an 
estimate  of  HPR  (Human  Performance 
Reliability)  and  the  performance  time 
associated  with  each  module  is  obtained 
with  their  mathematical  synthesis  provided 
as  an  estimate  of  the  complete  system's 
performance. 

The  problems  underlying  this  procedure 
include  assumptions  of  normal 
distributions,  of  a  single  distribution 
v  derlying  task-module  performance  time, 

.  '  of  interactions  among  task  modules. 

The  statistical  methods  used  to  analyze  the 
data  included  calculation  of  the  mean, 
median,  standard  deviation,  t-tests,  and 
Duncan's  Multiple  Range  Test. 

The  apparatus  for  this  study  included  a 
display/control  device,  two  books  of  Z 
tables  and  a  device  for  displaying  single 
lines  of  a  computer  print-out  on  a 
trial-by-trial  basis. 

The  Z-values  were  obtained  by  assessing  the 
tables  using  the  displayed  X  and  Y  values 
as  table  coordinates.  The  display/control 
device  consisted  of  five  analog  meters  and 
five  digital  read-out  modules;  one  meter 
was  designated  the  "Primary  Meter,"  the 
remaining  four  meters  displayed  extraneous 
information,  while  three  of  the  five 
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digital  modules  were  collectively  labeled 

"Y". 


5.3  The  following  results  were  indicated:  (1) 
the  normality  assumption  for  distribution 
of  task  time  is  inappropriate;  (2)  the 
rules  for  combining  task  tiroes  are 
satisfactory  if  the  underlying  distribution 
of  task  tiroes  is  known;  (3)  HPR  is  affected 
severely  by  combining  tasks;  and  (4)  any 
model  for  estimating  HPR  will  require 
parameters  to  account  for  task  combining 
and  difficulty. 
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Allocation  of  System  Effctiveness  Require¬ 
ments  (SER's)  is  the  process  of  determining 
how  the  total  system's  effectiveness  re¬ 
quirements  distribute  among  the  system's 
constituent  man-machine  functional 
units/states.  To  develop  a  procedure  for 
effectiveness  requirements  allocation, 
guidelines  must  be  generated  for: 

(1)  Specifying  the  system  effectiveness 
requirement  along  all  its  dimensions. 

(2)  Partitioning  the  system  into  segments 
and  states. 

(3)  Characterizing  and  specifying  input 
data. 

(4)  Relating  the  SER  to  system  segments 
consistent  with  the  input  data. 

The  concept  of  requirements  allocation 
implies  a  multiplicity  of  contributors 
to  the  meeting  of  those  requirements. 
"Contributors'1  generally  fall  into  one  of 
two  categories  of  verbal  description:  (1) 
activities,  or  (2)  system  states. 

The  author  found  that  analysts  who  are 
activity-oriented  tend  to  be  more 
stimulus-bound  and  less  free  from 
pre-conceived  notions  than  those  who  are 
state-oriented.  Specification  of  system 
states  tends  less  to  imply  transitionary 
methods  for  achieving  those  states  and 
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tends  to  lead  to  a  creative,  open-minded 
approach  to  analysis-both  for  new  designs 
and  for  evaluation  of  existing  systems. 

It  is  necessary  that  valid  system 
effectiveness  requirements  exist  and  are 
derived  from  mission  analyses,  and  that  the 
system  is  partitioned  into  manageable  units 
for  evaluation  of  their  contribution  to 
system  performance.  There  still  remains 
the  need  for  relevant  data  and  procedural 
rules  for  systematically  applying  those 
data  to  enable  allocation  of  the  given  SERs 
among  the  component  units. 

A  stipulated  value  for  the  SER  is  estab¬ 
lished  to  define  an  acceptable  level  of 
performance  with  respect  to  system 
objectives.  Effectiveness  requirements  may 
take  the  form  of  a  single  value,  several 
values  or  an  interval.  These  values 
represent  levels  of  effectiveness  which  are 
acceptable  under  specified  operating  or 
environmental  conditions.  When  more  than 
one  effectiveness  dimension  is  needed  to 
reflect  the  system  objective  adequately, 
the  SER  may  be  represented  as  an  index 
resulting  from  the  mathematical  combination 
of  values  on  several  effectiveness 
dimensions. 

For  allocation  of  SERs,  mission  analyses 
must  have  been  directed  toward  defining 
requirements  appropriate  for  effectiveness 
analyses.  Values  along  all  relevant 
dimensions  must  emerge  as  an  end  product. 
Currently  such  end  products  are  sorely 
lacking  due  to  the  intuitive  approach  to 
design  for  meeting  imprecisely  defined 
system  objectives.  SERs  are  rarely 
specified,  either  because  (1)  they  had  not 
b«en  considered,  or  (2)  the  researchers 
don't  wish  to  face  the  fact  that  serious 
objectives  may  not  always  be  reached,  or 
(3)  they  are  unwilling  to  record 
fallibility. 
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Quantification  of  effectiveness  requires 
identification  of  one  or  mere  measurement 
dimensions.  Most  freiuently  used 
measurement  dimensions  are  accuracy,  time, 
quantity,  and  rate,  constrained  by  cost 
limitations.  Effectiveness  dimensions  must 
be  related  as  directly  as  possible  to 
stated  system  objectives. 

The  allocation  of  SERs  must  provide  a  set 
of  performance  requirements  or  standards  at 
a  level  sufficiently  elemental  to 
facilitate: 

(1)  Trade-off  studies. 

(2)  Relative  appraisal  of  various  system 
design  concepts. 

(3)  Absolute  evaluation  of  given  design 
concepts. 

(i|)  Absolute  evaluation  of  a  given  design 
against  the  SERs  established  for  the 
system. 


Mitchell,  M.B.,  Smith,  R.L.  4  Blanchard,  R.E.  Test  appll cation  of  TEPPS  on  a 
Navy  CIC  subsystem.  Santa  Monica,  CA:  Dunlap  and  Assoc.,  Inc.,  August  1967. 
(AD-821  571). 
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Construct  the  Graphic  State  Sequence 
Model  (GSSM)  relating  system  required 
states  to  one  another  and  to 
man-machine  activities. 

Relate  probability  and  time  to  achieve 
system  output  to  probability  and  times 
to  accomplish  defined  man-machine 
activities : 

OR 

Employ  TEPPS  computer  program  for 
deriving  standards  by  allocating 


1.1  The  Technique  for  Establishing  Personnel 

Performance  Standards  (TEPPS)  provides  for 
establishing  performance  standards  that  are 
defined  as  criterion  measures  along  such 
effectiveness  dimensions  as  probability  of 
accomplishment  and  time  to  perform 
definable  man-involved  operations.  The 
technique  involves  analysis  of  a  system 
into  deseribable  and  useful  units  of 
personnel  behavior.  Once  that  has  been 
accomplished,  system  effectiveness 
requirements  are  allocated  along  the  units, 
and  the  resulting  standards  can  be  used  to 
evaluate  the  efficacy  and  efficiency  of  a 
particular  system  design  relative  to  stated 
system  effectiveness  requirements. 

To  achieve  those  objectives,  application  of 
TEPPS  involves  the  following  general 
procedures: 

(1)  Collect  and  organize  pertinent  data. 

(2) 


(3) 


6.  Further  Research  Areas 

6.1  Measurement  System 
Limitations 

6.2  Research  Potentials/ 
Priorities 

6.3  Research  Planning 


effectiveness  requirements  to 
nan-machine  activities: 

(2)  Compare  performance  standards  with 
human  capability  values. 

(3)  Evaluate  differences  and  determine 
optimal  corrective  action. 

2.1  A  field  test  of  the  TEPPS  was  performed  on 
a  simulated,  conventional  radar  detection 
and  record-keeping  subsystem  of  an  anti-air 
warfare  system. 

3.1  Data  was  collected  by  observation, 
documentation  and  interviews  at  a  training 
center,  and  a  GSSM  was  developed. 

3.2  Since  the  program  required  human  capability 
estimates,  subjective  data  were  collected 
from  16  expert  judges  on  29  unique 
activities  identified  in  the  GSSM. 

3.1*  Indices  of  probability  of  accomplishment 

and  performance  time  estimates  were  derived 
from  those  data  and  were  included  as  part 
of  the  input  to  TEPPS  computer  program, 
along  with  the  coded  GSSM  and  the  imposed 
effectiveness  requirements.  TEPPS  computer 
program  was  run  and  probability  and  time 
standards  were  established  under  all  modes 
of  system  operation. 
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The  objectives  of  the  project  were  to 
develop  a  methodology  with  a  set  of 
procedures  for  applying  decision/risk 
analysis  to  the  design  of  operational  tests 
and  the  analysis  of  operational  test 
results.  An  operational  test  was  defined 
as  that  test  and  evaluation  conducted  to 
estimate  the  prospective  system's  utility, 
operational  effectiveness,  and  operational 
suitability.  One  of  the  objectives  of 
operational  tecting  is  an  independent 
evaluation  of  competing  systems  resulting 
in  some  statement  of  relative  attributes 
and  preference. 

The  project  contains  a  review  of  four 
analytical  procedures  for  risk  assessment. 
Risk  analysis  can  be  viewed  as  the  process 
of  combining  the  risk  assessment  with 
alternative  courses  of  action  in  an 
iterative  cycle. 

(1)  "An  application  of  Multivariate 
Discriminant  Analysis  and 
Classification  Procedures  to  Risk 
Assessment  in  Operational  Testing." 
This  research  developed  a  methodology 
for  determining  an  index  useful  in  the 
assessment  of  risk  in  operational 
testing.  The  risk  assessment  problem 
examined  is  that  of  preference 
statements  regarding  systems.  To 
evalute  the  competing  systems, 
multivariate  distributions  of  each 
system  and  the  overlap  of  these 
distributions  is  used  to  determine  the 
index  of  risk.  The  index  of  risk  is  a 
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measure  of  similiarity  of  the 
competing  systems. 

(2)  "An  Application  of  Simulation 

Networking  Techniques  in  Operational 
Test  Design  and  Evaluation."  Opera¬ 
tional  tests  contain  several  activi¬ 
ties,  subtests,  or  subprograms.  Each 
of  these  activities  or  subtests  has 
related  functional  values.  These 
values  may  te  deterministic, 
stochastic,  or  some  mathematical 
transformation  of  a  value  computed  in 
an  earlier  activity.  These  activities 
lead  to  milestones  or  events  and  the 
outcome  of  the  operational  test  can  be 
represented  by  a  set  of  successful  and 
a  set  of  unsuccessful  events.  This 
set  of  conditions  describes  a 
stochastic  network.  Of  the  network 
analysis  tools,  network  simulation 
affords  the  greatest  versatility  and 
flexibility  in  modeling  this  set  of 
conditions.  Of  the  family  of  network 
simulation  programs  two  programs  have 
evolved  as  useful  analysis  tools  to 
assess  risk,  SOLVNET  and  VERT.  The 
objective  of  this  research  is  to 
investigate  the  use  of  these  network 
simulation  programs  in  the  design  of 
operational  tests  and  the  analysis  of 
operational  test  data. 

(3)  "An  Application  of  Bayesian  Analysis  in 

Determining  Appropriate  Sample  Sizes 
for  Use  in  U.S.  Army  Operational 
Tests."  The  research  was  devoted  to 
modifying  the  Bayesian  Techniques 
associated  with  determining  the 
minimum  sample  size  required  to 
construct  interval  estimates  of  the 
true  mean  of  an  experimental  or 
sampling  process  which  is  modeled  by  a 
normal  distribution  with  unknown 
parameters.  The  procedure  considers 
only  the  case  where  the  prior 
information  can  be  represented  by  a 
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normal  distribution  with  known  mean 
and  known  variance. 

(4)  "Finding  a  Minimum  Risk  Path  through  a 
Network  Using  Resource  Allocation 
Techhiques."  The  objective  of  this 
research  was  to  develop  an 
optimization  method  for  network  risk 
analysis  where  a  resource  constraint 
is  present.  (This  thesis  was 
unfinished  and  only  the  methodology 
outline  was  presented.) 


Mumford,  J.  4  Smith,  J.P.  The  development  of  performance  criteria,  for 
turret  mechanics  (HumRRO  Research  Memo.).  Alexandria,  VA:  Human  Resource 
Research  Organization,  July  1961.  (AD-477  647). 
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2.1  The  armor  turret  mechanic  and  his  equipment 
in  the  training  environment  was  the  subject 
of  this  study. 

3.4  This  study  attempts  to  develop  performance 
criteria  for  turret  mechanics. 
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4.5  The  first  step  is  to  collect  information  on 
the  task  at  the  organizational  level.  This 
was  accomplished  by  studying  job  descrip¬ 
tions  and  interviewing  consultants 
knowledgeable  in  the  field. 

Tasks  selected,  in  a  large  measure, 
reflected  juogment  of  personnel  consulted. 
Exercises  and  testn  were  developed  and 
administered  to  subjects.  A  scoring  system 
was  developed  which  enabled  the  tests  to 
distinguish  degrees  of  adequacy  or 
inadequacy  of  performance.  On  the  basis  of 
this  testing,  the  exercises  were  revised 
and  a  new  draft  prepared. 
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Contextual  Components 
of  the  Process 

2.1  System  Definition 

2.2  Mission  Definition 
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5.3  Findings  Interpretation 
5. A  Conclusions  and 
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The  purpose  of  this  study  is  to  propose  a 
research  methodology  which  considers  the 
physical,  environmental,  and  decision¬ 
making  aspects  of  complex  service  systems 
as  a  total  system  and  to  present  a  methodol 
ogy  in  generalized  concepts  and  terminology 
to  emphasize  the  potential  application  to 
all  systems. 

A  schematic  diagram  is  presented  to  struc¬ 
ture  the  author's  view  of  the  physical 
service  system,  intra-system  decision 
making  and  the  environment. 

An  illustration  of  the  methodology  proposed 
is  also  presented  in  this  report.  The 
example  given  is  a  simple  production/inspec 
tion  system  producing  a  single  product  to 
inventory. 

The  outline  and  description  of  the  proposed 
research  methodology  is  presented  below: 

METHODOLOGY  OUTLINE 

(1 )  Identification  and  Modeling 

a.  Modeling  of  the  physical  system. 

b.  Characterization  of  the  system 
decision  process. 

c.  Modeling  of  the  system  criterion 
function. 

d.  Modeling  of  the  environmental 
response  process. 
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(2) 

Experimentation 

a. 

Input-output  analysis. 

b. 

Equilibrium  analysis. 

c. 

System  performance  evaluation. 

d. 

Variation  of  system-environment 
decision  variables. 

e. 

Variation  of  system  control 
variables. 

f. 

Variation  of  system  design 
variables. 

(3) 

Implementation 

An  illustration  of  the  process  is  presented 
which  includes  continuing  observation  of 
the  operating  system  and  its  environment, 
and  modification  of  the  system  model  as 
well  as  continuing  experimentation  of  the 
decision  algorithm. 

It  is  expected  that  field  studies  may 
present  far  more  of  a  challenge  because  of 
the  identification,  modeling  and  data 
requirements  which  will  arise  in  actual 
situations.  The  methodological  plan 
proposed  here  focuses  attention  on  these 
requirements  as  an  integral  part  of  systems 
analysis. 
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1.3  In  an  effort  to  improve  training  perform¬ 
ance  information,  this  study  is  directed 
towards  a  systematic  definition  of  perform¬ 
ance  and  development  of  methods  for  measure¬ 
ment.  The  primary  goal  is  to  provide  usa¬ 
ble  measurement  tools  for  attacking  prob¬ 
lems  related  to  combat-crew  training.  This 
report  covers  the  first  phase  -  -  defini¬ 
tion  of  requirements  for  information  based 
on  data  collection  surveys  to  six  selected 
combat-crew  training  sites. 

2.1  The  systems  addressed  were  the  A-7,  B-52, 
C-130,  C— 141,  F-4,  and  F-106  weapon  sys¬ 
tems,  their  crews,  and  their  training. 
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The  performance  of  trainees  during  and  at 
the  end  of  training  is  the  measure  by  which 
these  training  programs  can  be  judged. 

Instructional  system  development  requires 
that  performance  standards  are  identified 
so  that  the  most  efficient  approach  is  used 
to  train  for  the  needed  skills  and  knowl¬ 
edge  to  the  desired  level  of  performance. 

The  first  phase  of  the  study  consisted  of 
three  tasks:  (1)  Data  collection;  (2) 
Analysis  of  common  measurement 
requirements;  (3)  Dimensions  of  measurement 
modularity. 
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(1)  Data  Collection  -  All  existing 

subjective  and  objective  methods  of 
measuring  performance  during  and  at 
the  end  of  training  were  examined. 
Researchers  attempted  to: 

(a)  Properly  consider  measurements  in 
the  context  of  combat-crew 
training. 

(b)  Assess  the  measurements  already 
included  as  well  as  potential 
measurement. 

(c)  Assess  the  constraints  placed  by 
the  combat-crew  training 
environment  on  feasible  usable 
measurement  systems. 

(2)  Training  Measurement  -  -  At  each  site, 
information  was  collected  with  respect 
to: 

(a)  The  training  sequence. 

(b)  Points  where  measurement  exists. 

(c)  Measurement  possibilities. 

(d)  Feasibility  of  research 
measurement. 

(e)  Specific  new  measurement 
development. 

Six  quite  different  aircraft  were 
included  in  the  sample  for  measurement 
analysis  and  an  attempt  was  made  to 
determine  whether  a  simple  practical 
measurement  system  for  all 
applications  is  possible. 

(3)  Prototype  Measurement  -  -  As  a  natural 
extension  of  the  considerations  of 
measurement  commonality,  examples  of 
the  information  required  for  training 
were  developed.  These  measurement 
requirements  are  extensive  and 


complex.  However,  this  detail  is 
probably  needed  to  support  the 
training  process. 
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This  report  describes  the  second  phase  of  a 
study  directed  towards  a  systematic 
definition  of  performance  and  development 
of  methods  for  measurement  in  an  effort  to 
improve  training  performance  information. 
This  phase  deals  with  the  requirements  for 
a  measurement  system  to  process  the 
measurement  which  has  been  dictated  by  the 
previous  reports. 

Research  procedures  were  developed  to 
indicate  the  operation  in  which  a 
measurement  system  is  to  serve  as  a  tool  in 
achieving  research  goals,  and  methods  of 
measurement  processing  were  determined  to 
investigate  the  nature  of  data  processing 
associated  with  training  research 
measurement.  In  addition,  system  criteria 
to  guide  design  tradeoffs  were  addressed 
and  preliminary  systems  analyses  conducted 
to  establish  measurement  system 
requirements  which  follow  rather  directly 
from  the  system  criteria.  In  the  latter 
effort,  measurement  parameters  were 
identified  and  the  approximate  number  of 
measurement  parameters  for  each  flight 
phase  were  presented. 
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This  report  describes  Phase  IIIA  of  an 
effort  to  improve  training  performance 
information.  This  study  is  directed 
towards  a  systematic  definition  of 
performance  and  development  of  methods  for 
measurement.  The  primary  goal  of  the  study 
is  to  provide  usable  measurement  tools  for 
attacking  problems  related  to  combat-crew 
training.  In  accordance  with  the  initial 
definition  of  this  study,  emphasis  was 
placed  on  pilot  performance,  but  it  was 
soon  recognized  that  avoiding  the 
contribution  of  other  crew  members  and  the 
interaction  between  crew  members,  had 
serious  consequences.  Therefore, 
additional  tasks  were  undertaken. 

The  systems  addressed  in  this  phase  are  the 
F4E  and  the  C1H1A  weapon  systems  and  their 
combat  crew  training  units. 

After  the  requirements  for  pilot 
proficiency  measurements  were  established, 
time  was  given  to  the  consideration  of  the 
variety  of  possible  systems  and  to  the 
constraints  imposed.  A  conceptual  design 
had  been  formulated,  consisting  of  feasible 
alternatives,  to  indicate  the  type  of 
information  possible,  the  places  where  such 
Information  would  be  useful,  and  the 
possible  ways  such  information  could  be 
collected.  It  was  concluded  that 
measurement  of  crew/system  performance  (as 
opposed  to  pilot  only)  was  important  to  a 
thorough  description  of  performance  for 
certain  tasks  and  missions.  Consequently, 
the  additional  task  of  defining  crew 
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proficiency  measurement  requirements  was 
undertaken  and  these  requirements  will  be 
incorporated  into  the  conceptual  design. 

4.5  This  phase  addressed  the  problem  of  crew 
performance  as  opposed  to  pilot 
performance.  A  survey  was  conducted  to 
thoroughly  define  requirements  in  the 
measurement-system  to  sense  crew 
interaction.  The  survey  was  conducted  at 
nine  sites,  two  of  which  were  visited 
specifically  for  the  collection  of 
information  on  crew  performance. 

5.4  One  of  the  main  difficulties  in  performing 
detailed  analyses  of  crew  interaction  is 
the  lack  of  specificity  in  stating  require¬ 
ments  for  crew  relationships.  Since  crew 
interaction  techniques  are  non-standard, 
and  specific  techniques  are  non-trained, 
measurement  of  crew  interaction  cannot  be 
explicitly  defined  in  most  cases.  A 
further  difficulty  is  the  degree  to  which 
the  performance  of  each  crew  member  is 
dependent  upon  the  performance  of  others. 

It  was  concluded  in  this  phase  of  the  study 
that  better  methods  of  crew  interaction 
must  be  discovered  and  developed. 
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1.3  This  report  covers  a  further  phase  in  the 

research  for  the  improvement  of  combat-crew 
training.  This  effort  deals  with  design 
studies  to  determine  desirable  system 
features  to  meet  the  research  needs 
documented  in  the  earlier  reports  of  this 
sequence. 


2.  Contextual  Components 
of  the  Process 

2.1  System  Definition 

2.2  Mission  Definition 

2.3  Environment  Definition 

2.4  General  Constraints 

2.5  Performance 
Requirements,  Ultimate 


An  analysis  of  factors  to  be  considered  in 
training  measurement  system  design  is 
presented  and  the  nature  of  tradeoffs  for 
each  system  criterion  established  are 
indicated. 


2.6  Performance 

Criteria,  Ultimate 

Analytic  Components 
of  the  Process 


3.1 

Practical  Measurable 
Attributes 

(1) 

3-2 

Practical  Attribute 

Measures 

(2) 

3-3 

Performance 

Requirements,  Specific 

3.4 

Performance 

Criteria,  Specific 

(3) 

3.5 

Measurement  Procedures 

4. 

Planning  Components 
of  the  Process 

(4) 

4.1 

Analytic  Methods 

(5) 

4.2 

Parameter  Determinations 

4.3 

Apparatus  for  Testing 

4.4 

Personnel  for  Testing 

(6) 

4.5 

Test  Plans 

5. 

Application  Components 
of  the  Process 

(7) 

5.1 

Test  Execution 

5.2 

Dmta  Analysis 

5.3 

Findings  Interpretation 

(8) 

5.4 

Conclusions  and 
Recommendations 

3.1  A  number  of  analyses  were  performed  in  this 
program  to  guide  design  decisions.  Eight 
primary  analytic  steps  were: 

(1)  Determine  Measurement  Needs 


Measurement 


Sufficiently  Accurate 
(8)  Determine  Cost  Tradeoff  Data 


6.  Further  Research  Areas 
6.<  Measurement  System 
Limitations 

6.2  Research  Potentials/ 
Priorities 

6.3  Research  Planning 
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A  measurement  example  was  produced  as  a 
product  of  visits  to  combat-crew  training 
units  to  attempt  to  express  the  main  items 
of  information  relevant  to  training.  Using 
this  as  a  stimulus,  preliminary  measurement 
definitions  were  made  along  with  assumed 
techniques  for  computation,  leading  to 
identification  of  a  required  set  of 
parameters  for  measurement  generation. 

These  analyses  were  begun  in  earlier 
program  phases,  and  subsequently  were 
revised  and  extended. 

Continuing  from  the  basic  analyses, 
specific  computational  algorithms  were 
chosen  for  both  automatic  and  manual  modes 
of  measurement,  forming  an  initial  software 
specification.  The  next  steps  in  the 
sequence  attempt  to  determine  best  methods 
for  sensing  the  needed  information  and  the 
nature  of  appropriate  data  processing 
equipment.  As  video  or  photographic  means 
of  sensing  information  must  be  considered, 
it  follows  that  a  minimum  resolution  for 
such  devices  must  be  specified  to  ensure 
that  the  desired  data  are  sufficiently 
legible.  Lastly,  data  are  collected 
relating  to  cost  and  personnel  requirements 
to  permit  tradeoff  analyses  between 
alternative  measurement  system  candidates. 

The  design  analyses  resulted  in  tradeoff 
comparisons  at  two  levels:  (1) 
comparison  of  competing  data  sources,  i.e., 
audio,  X-Y,  video/ photo,  and 
instrumentation  (digital  recording),  and 
(2)  comparison  of  systems  built  around  only 
video/photo  sensors  and  only  digital 
recording.  Tradeoff  comparisons  at  the 
first  level  reveal  the  rule  of  alternative 
data  sources,  while  second-level 
comparisons  establish  cost-effective  system 
combinations. 
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5.4  A  clear  uncomplicated  choice  is  not 

possible  between  video/photo  and  digital 
recording  approaches  to  measurement  system 
design,  but  if  s  <ch  a  choice  must  be  made, 
video/photo  recording  will  be  chosen  for 
cost,  information  provided,  flexibility  and 
ease  of  use. 

However,  a  hybrid  system,  combining  the 
advantages  of  both,  is  preferable  to  either 
type  of  recording  alone.  Due  primarily  to 
cost,  the  bulk  of  measurement  parameters 
would  be  derived  from  a  video/phc  ',o  system, 
and  the  remainder  with  a  small  digital 
recording  capability.  It  would  be 
desirable  for  the  major  components  of  a 
hybrid  system  to  have  a  stand-alone 
capability  of  modest  means  and  power  for 
all  combat-crew  training  measurement  when 
used  together.  Auditory  data  recording 
should  be  incorporated  together  with  the 
option  for  merging  data  with  that  from 
ground-tracking  radar.  All  data  recording 
must  include  provision  for  synchronization 
with  other  data  sources. 
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4.5  Test  Plans 
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5.1  Test  Execution  ‘ 
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5.3  Findings  Interpretation 
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Phase  HID  of  this  study,  designed  to 
improve  training  performance  information, 
presents  specifications  and  an  imple¬ 
mentation  plan  for  the  performance 
measurement  system  recommended  as  a  result 
of  this  program.  The  report  covers  the 
acquisition  and  processing  of  data 
personnel  and  facilities  required,  and 
describes  the  implementation  plan. 

The  appendix  of  this  report  provided  a 
tentative  survey  and  specification  of 
equipment  and  instrumentation  required  for 
the  implementation  of  the  performance 
measurement  program.  There  are  three  major 
sections  to  this  appendix.  Section  one 
addresses  the  monitoring  and  data 
collection  equipment  anticipated  for 
installation  on  the  test  aircraft(s).  The 
second  section  discusses  the  post-flight 
ground  debriefing  facilities  and  associated 
equipment.  The  last  section  describes  the 
data  processing  facilities  and  equipment 
requirements  anticipated  for  detailed 
analysis  and  evaluation  of  collected  data. 

An  analysis  of  personnel  requirements  was 
conducted  in  Phase  IIIC  of  this  study. 
Further  clarification  is  presented  here 
with  a  reassessment  of  the  total  personnel 
requirements  of  the  performance  measurement 
system. 


6.  Further  Research  Areas 

6.1  Measurement  System 
Limitations 

6.2  Research  Potentials/ 
Priorities 

6.3  Research  Planning 
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*1.5  The  implementation  plan  recommended  was 

based  on  Air  Force  Systems  Command  Manual 
AFSCM  375-5.  There  are  five  major  steps 
required  to  implement  this  plan: 

(1)  Selection  of  a  system  integration 
contractor . 

(2)  Completion  of  preliminary  detail 
system  and  subsystem  design. 

(3)  Selection  of  the  final  system  design 
with  appropriate  testing. 

(4)  Procurement  of  system  hardware  and 
system  integration. 

(5)  Completion  of  final  system  tests 
resulting  in  system  turnover  to  the 
Air  Force. 


to  the  test  and  evaluation  phase  of  the  acquisition  of  a  major  air  system 
(TR-77-3) .  McLean,  VA:  Decisions  and  Designs,  Inc.,  April  1977.  ( AD- AO 40 
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1.1  Currently  the  procedure  for  evaluating  the 
suitability  of  the  air  system  from  a  human 
engineering  standpoint  consists  of  test 
pilot  inspections  of  the  air  system.  The 
deficiencies  are  noted,  the  report  is 
written,  and  the  defect  put  into  one  of 
three  categories.  However,  time  is 
important  and  so  a  method  is  needed  to 
communicate  the  information  quicker  with  a 
prioritization  of  corrections  as  well. 

1.2  The  answer  to  the  questions  of  time  and 
prioritization  is  a  computerized  system 
containing  the  test  and  evaluation 
information  appropriately  prioritized.  One 
task  to  this  goal  is  the  development  of  a 
hierarchical  evaluation  structure  which 
relates  all  the  test  and  evaluation 
information  to  the  missions  of  the  FI 8  (the 
aircraft  system  under  consideration). 

A  diagram  of  that  hierarchy  is  presented  in 
the  report. 
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4.2  Parameter  Determinations 

4.3  Apparatus  for  Testing 

4.4  Personnel  for  Testing 

4.5  Test  Plans 


5.  Application  Components 
of  the  Process 

5.1  Test  Execution 

5.2  Data  Analysis 

5.3  Findings  Interpretation 

5.4  Conclusions  and 
Recommendations 

6.  Further  Research  Areas 

6.1  Measurement  System 
Limitations 

6.2  Research  Potentials/ 
Priorities 

6.3  Research  Planning 
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5.1  Test  Execution 
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The  present  document  deals  with  the  attempt 
to  design  a  guidebook  for  the  evaluation  of 
man-machine  systems.  The  researchers  feel 
that  advocating  the  rise  of  the  information 
processing  data  and  modeling  literature  as 
a  base  presupposes  that  models  will  be 
developed  with  the  most  elemental  compo¬ 
nents  of  performance  and  building  from 
there . 

The  limitations  of  models  that  were  derived 
by  human  information  processing  specialists 
are: 

(1)  The  models  tend  to  be  compartmental¬ 
ized  by  the  very  fact  that  most  of 
them  deal  with  particular  stages  of  in¬ 
formation  processing  rather  than  being 
integrative  of  human  performance  in 
general. 

(2)  For  the  most  part,  human  information 
processing  models  deal  with  the 
average  performance  of  well-motivated, 
highly  practiced  individuals,  under 
relatively  ideal  conditions.  There 
are  many  hypotheses  but  few  data  and 
virtually  no  models  in  the  information 
processing  literature  on  how  human 
performance  capacities  change  under 
stress,  under  reduced  motivation, 
before  practice  has  stabilized 
performance,  when  interacting  in 
groups  or  on  the  range,  or  characteris¬ 
tics  of  individual  differences. 


6.  Further  Research  Areas 

6.1  Measurement  System 
Limitations 
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6.3  Research  Planning 
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6.2  Recommendations  for  further  research  and 
development  of  large  scale  systems 
modelling  efforts  include  the  following: 

(1)  Development  of  a  test  bed  to  evaluate 
alternative  model  formulations  of 
common  task  environments  and  to 
conduct  empirical  validation  studies 
to  compare  model  predictions  with 
actual  human  peformance. 

(2)  Methodological  research  on: 

a.  Implications  of  combining 
sub-ta3k  or  information 
processing  component  models  on 
system  performance  in  the 
aggregate. 

b.  Validation  of  large  scale 
simulation  models. 

c.  Development  of  guidelines  for  the 
acceptable  number  of  free 
parameters  in  useful  predictive 
models. 

(3)  Further  model  development  in  topical 
areas  of  high  priority  for  representa¬ 
tion  of  command  and  control  systems. 

(4)  Advancing  to  state-of-the-art  with 
respect  to  the  specific  modeling 
approaches  discussed  in  the  body  of 
the  report. 


A-338 


Phatak,  A.V.  Improvement  In  weapon  system  effectiveness  by  application  of 
Identification  methods  for  determining  human  operator  performance  decrements 
under  stress  conditions  (AMRL-TR-73-38) .  Palo  Alto,  CA:  Systems  Control, 
Inc.,  December  1973.  (AD-773  856). 


1  Topics  Relevant 

1  1 

1  1 

1  to  System  Development 

i Topic! 

land  Evaluation  Technology 

i  No.  ! 

ABSTRACT 

1.  State  of  the  Art  Review  1*3 

of  the  Process 

1 . 1  General  System 
Measurements 

1.2  System  Taxonomy 
Model  (STM) 

1.3  Overall  Conceptual 

Process  Model  (CPM)  2,1 

2.  Contextual  Components 
of  the  Process 

2.1  System  Definition  2  2 

2.2  Mission  Definition 

2.3  Environment  Definition 

2.4  General  Constraints 

2.5  Performance 
Requirements,  Ultimate 

2.6  Performance  J’1 

Criteria,  Ultimate 

3.  Analytic  Components 
of  the  Process 

3.1  Practical  Measurable 
Attributes 

3-2  Practical  Attribute 
Measures 

3-3  Performance 

Requirements,  Specific 

3.4  Performance  4,1 

Criteria,  Specific 

3.5  Measurement  Procedures 

4.  Planning  Components 
of  the  Process 

4.1  Analytic  Methods 

4.2  Parameter  Determinations 

4.3  Apparatus  for  Testing 

4.4  Personnel  for  Testing 

4.5  Test  Plans 

~5.  Application  Components 

of  the  Process 
5.1  Test  Execution 
5-2  Data  Analysis 

5.3  Findings  Interpretation 

5.4  Conclusions  and 
Recommendations 


The  problem  studied  in  this  report  is  the 
development  of  realistic  models  for  weapon 
system  controllers  that  can  be  used  to 
predict  the  effectiveness  of  manned  weapon 
systems  under  stress  conditions. 

The  system  chosen  for  this  study  was  the 
pilot  of  a  high  performance  aircraft. 

The  aircraft  missions  were  engagements  in 
either  precision  weapon  delivery  or 
air-to-air  combat. 

The  task  chosen  for  this  experiment  was  a 
single  axis  stable  tracking  task  whose 
dynamics  were  representative  of  a  high 
performance  aircraft  pitch  axis  control 
system.  However,  two  other  compensatory 
tracking  tasks  were  also  employed,  but  only 
the  task  described  above  is  considered  in 
this  report. 

Two  types  of  models  were  considered:  (1) 
the  input-output  stochastic  linear 
state-variable  models  (equivalent  to 
describing  function  models  when  process 
noise  _=  0),  and  (2)  the  optimal  control 
model  developed  by  Kleinman,  et  al.  The 
maximum  liklihood  identification  technique 
was  used  in  estimating  model  parameters 
from  input-output  data. 


6.  Further  Research  Areas 

6.1  Measurement  System 
Li  dilutions 

6.2  Research  Potentials/ 
Priorities 

6.3  Research  Planning 


A-339 


Topics  Relevant 
to  System  Development 
and  Evaluation  Technolo 


Topic 

No. 


The  subjects  performed  the  tasks  at  sea 
level  followed  by  the  same  task  at  a 
simulated  altitude  of  either  12,000  or 
20,000  feet.  Each  run  of  this  experiment 
consisted  of  two  tracking  periods.  Each 
period  was  preceded  by  one  minute  of 
pre-breathing  at  the  indicated  altitude 
followed  by  one  minute  of  tracking. 

The  order  of  presentation  of  the  simulated 
altitudes  and  tasks  was  randomized  in  order 
to  minimize  the  effects  of  learning  and 
anticipation  of  experimental  factors. 

Results  showed  that  the  identification 
algorithm  was  highly  successful  in 
identifying  the  parameters  of  the  stochas¬ 
tic  state  variable  models.  However, 
difficulties  were  encountered  in  applying 
the  technique  to  identifying  parameters  of 
the  optimal  control  model.  It  appears  that 
the  unsatisfactory  identification  results 
with  the  optimal  control  model  are  due, 
first,  to  the  over  parameterization  of  the 
model  structure  and  second,  because  of  the 
finite  data  and  specific  input-output  data. 
Inferences  regarding  performance  decrements 
due  to  stress  were  precluded  because  of 
insufficient  data  base  for  this  study. 
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1.1  In  this  report,  the  optimal  control  model 
is  analyzed  from  the  system  identification 
point  of  view  to  determine  which  parameters 
can  be  identified.  As  a  result,  a  proced¬ 
ure  for  the  identification  of  the  optimal 
control  model  parameters  from  measured 
experimental  data  has  been  developed.  This 
procedure  is  validated  by  application  to 
experimental  human  operator  data  provided 
by  Aerospace  Medical  Research  Laboratory. 

In  addition,  a  systematic  approach  is 
presented  towards  the  development  of  a 
metric  for  measuring  and  ranking  system 
difficulty  as  experienced  by  human 
operators  in  terms  of  system  properties 
such  as  the  degree  of  controllability  and 
the  degree  of  observability. 
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1.3  This  report  presents  a  checklist  to  be  used 
for  human  engineering  acceptance  testing 
and  design  verification.  It  is  also  to  be 
used  as  a  human  engineering  guide. 

This  checklist  is  a  device  by  which  human 
engineering  design  criteria  and  standards, 
as  set  forth  in  MIL  STD-803A-1,  can  be 
verified  during  acceptance  testing  and 
checkout. 

The  human  engineering  design  is  verified 
(a)  in  plant  during  equipment  acceptance 
testing  and  (b)  in-the-field  during 
subsystem  and  system  checkout. 


3.  Analytic  Components 
of  the  Process 

3.1  Practical  Measurable 
Attributes 

3.2  Practical  Attribute 
Measures 

3.3  Performance 
Requirements,  Specific 

3.4  Performance 
Criteria,  Specific 

3-5  Measurement  Procedures 
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4.1  Analytic  Methods 
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4.4  Personnel  for  Testing 

4.5  Test  Plans 


Plant  acceptance  testing  is  composed  of  two 
parts  -  -  visual  inspection  and  functional 
test.  Field  Con-site)  checkout  has  five 
parts:  completion  of  checks  to  verify 
compliance  with  MIL-STD-803A-1  on  equipment 
acceptance-tested  in-plant;  inspection  and 
test  of  all  control/display  equipment; 
monitoring  of  subsystem  functional  tests; 
monitoring  of  system  functional  tests 
during  fly-bys  and  during  system  checkout 
with  the  checkout-subsystem;  analysis, 
evaluation,  and  documentation  of 
verification  results. 
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The  System  Effectiveness  Analyzer  (SEA)  is 
a  weapon  system  measuring  system  which  is 
based  primarily  upon  the  utilization  of 
existing  equipment  (controllers  and  periph¬ 
erals)  supplemented  by  the  additional  soft¬ 
ware  required  to  enhance  existing  capabili¬ 
ties  to  evaluate  simulated  weapon  systems. 

The  SEA  System  has  two  major  functions: 

(1)  to  checkout,  control,  monitor,  and 
perform  statistical  analyses  associated 
with  tracking  simulators;  (2)  to  provide  an 
estimation  of  weapon  system  effectiveness. 

There  were  several  design  features  that 
were  considered  imperative  to  the  develop¬ 
ment  of  the  SEA  measurement  system:  input 
flexibility,  operator  input  options,  simula¬ 
tor  interface,  turn-around-time  minimiza¬ 
tion,  and  expansion. 

The  weapons  system  includes  the  human 
tracker  as  its  principal  sensor  of  target 
motion.  The  metrics  used  to  evaluate 
weapon  system  performance  were  the  weapon's 
round-by-round  probability  of  kill  and  the 
resulting  target's  probability  of  survival. 
Angular  error  statistics  are  used  to 
evaluate  tracker  performance.  One  SEA 
simulation  has  been  Implemented.  The 
simulated  weapon  system  is  an  AAA  battery 
using  manned  optical  sights.  The  target 
being  engaged  is  an  aircraft  deploying 
optical  countermeasures.  The  SEA  commands 
the  tracking  simulator's  display  of  the 
sights  picture,  both  with  regard  to  target 
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position  and  countermeasure  stress  level. 
Outputs  from  the  simulator  include  the 
subject's  control  movements  and  the  gun 
trigger  state.  These  outputs  are  used  by 
the  SEA,  in  conjunction  with  the  target 
trajectory,  to  update  the  sights  picture, 
to  accumulate  the  statistical  measures  of 
the  tracker's  performance,  and  to  compile 
the  data  on  which  to  base  the  effectiveness 
analysis  of  the  AAA  system.  The  collected 
data  are  essentially  that  which  would  have 
been  determined  from  an  actual  AAA  system. 
The  Lead  Angle  Computer  directs  the  gun  to 
the  predicted  position  of  the  aircraft. 

The  mean  miss  distance  of  each  shell  fired 
is  estimated  and  error  sources  appropriate 
to  the  gun  system  are  used  to  find  the 
distribution  of  shot  relative  to  the  target 
aircraft.  Weapon  lethality  and  target 
vulnerability  data  are  then  used  in  con¬ 
junction  with  the  shell  mean  miss  distance 
and  scatter  area  to  determine  the  kill 
probability  of  each  shell.  The  probability 
that  the  aircraft  survives  the  mission  is 
derived  after  each  round  is  fired. 
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1.1  For  several  years,  measurement  and  instruc¬ 
tional  specialists  have  been  drawing  dis¬ 
tinctions  between  norm-referenced  and 
criterion-referenced  approaches  to  measure¬ 
ment.  A  norm-referenced  test  identifes  an 
individual's  performance  in  regards  to  the 
performance  of  others;  a  criterion-refer¬ 
enced  test  identifies  performance  in  regard 
to  an  established  standard  of  performance. 
This  report  examines  the  implications  of 
these  two  approaches  to  measurement,  partic¬ 
ularly  criterion-referenced  measurement, 
with  respect  to  variability,  item  con¬ 
struction,  reliability,  validity,  item 
analysis,  reporting  and  interpretation. 
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1.  State  of  the  Art  Review 
of  the  Process 

1 . 1  General  System 
Measurements 

1.2  System  Taxonomy 
Model  (STM) 

1.3  Overall  Conceptual 
Process  Model  (CPM) 

2.  Contextual  Components 
of  the  Process 

2.1  System  Definition 

2.2  Mission  Definition 

2.3  Environment  Definition 

2.4  General  Constraints 

2.5  Performance 
Requirements,  Ultimate 
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1.1  This  report  contains  16  human  factors 

testing  techniques  contributed  by  a  number 
of  Air  Force  and  contractor  personnel. 
While  limited  in  number,  it  covers  a  broad 
segment  of  the  human  factors  testing 
spectrum.  This  catalog  was  developed  as  a 
prototype  to  determine  its  usefulness  to 
human  factors  personnel  as  a  reference 
source  for  human  factors  testing  devices 
and  techniques. 

The  following  information  is  generally 
provided  on  each  technique. 

(1)  Purpose 
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3.4  Performance 
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(2)  Author 

(3)  Published  references 

(4)  Description  of  technique,  including 
how  it  is  utilized,  at  what  phase  in 
the  system  it  is  most  useful,  and 
what  kind  of  equipment  is  needed 
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4.1  Analytic  Methods 

4.2  Parameter  Determinations 

4.3  Apparatus  for  Testing 

4.4  Personnel  for  Testing 

4.5  Test  Plans 


(5)  The  advantages  and  disadvantages  of 
the  technique 

(6)  The  limitations,  validity  and  relia¬ 
bility  of  the  technique 


5.  Application  Components 
of  the  Process 

5.1  Test  Execution 

5.2  Data  Analysis 

5.3  Findings  Interpretation 

5.4  Conclusions  and 
Recommendations 


The  human  factors  techniques  included  in 
this  report  and  a  description  of  their  pur¬ 
pose  are  presented  below. 

(1)  Miniature  Simulation 
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To  determine  the  maintainability  of 
an  aircraft  by  evaluating  the  design 
of  the  aircraft  and  its  associated 
aerospace  ground  equipment  through 
the  use  of  a  miniature  scale  model. 
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(2)  Automated  Readability  Index 

To  determine  reading  difficulty  level 
derived  from  ratios  representing  word 
difficulty  and  sentence  difficulty. 

(3)  Flight  Simulator  and  Associated 
Simulation  Complex 

To  provide  engineering  data  in  the 
area  of  guidance  and  control,  naviga¬ 
tion,  human  factors,  target  acquisition, 
trajectory  error  analysis,  sensor  and 
display  development,  and  flying  qualities. 

(4)  Terrain  Visibility  Definition 

To  determine  the  in-flight  vertical  and 
horizontal  visual  angles.  This  data 
can  then  be  used  to  determine  the  visual 
area  available  to  aircrews  with  the  air¬ 
craft  in  the  horizontal  flight  attitude. 

(5)  Definition  of  Functional  Eye  Position 

To  determine  the  eye  position  of  pilots 
seated  in  a  cockpit  in  their  normal 
flight  posture.  This  technique  can  be 
used  to  generate  a  three-dimensional 
eye  ellipse  with  subjects  seated  either 
in  a  cockpit  mockup  or  in  an  actual 
aircraft. 

(6)  Definition  of  Dynamic  Body  Positions 

To  locate  various  body  components  while 
a  man  is  riding  a  captive  ejection  seat. 

(7)  Personnel  Activity  Analysis  Radio 
System  (PAfiRS) 

To  collect  job  activity  information, 
via  radio,  on  a  number  of  operational 
or  maintenance  personnel  while  they 
are  working  dispersed  over  a  wide  area. 

(8)  Work  Station  Analyzer 

To  determine  the  viewing  angles  and 
distances  of  display  panels. 
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(9)  Secondary  Task  Monitoring 

To  determine  performance  decrement 
under  dynamic  conditions  on  a  flight 
simulator. 

(10)  Operator  Analysis  in  Command  and 
Control  Systems 

To  analyze  those  operator  performance 
sequences  which  are  judged  to  be  the 
most  difficult  to  perform  without 
error. 

(11)  Human  Factors  Test  Planning 

To  describe  the  development  of  a  test 
plan  for  field  evaluation.  It  in¬ 
cludes  the  development  of  test  objec¬ 
tives  and  determination  of  data 
requirements  to  plan  a  test  program 
which  gives  a  well  balanced  coverage 
of  the  areas  of  interest. 

(12)  Induced  Error  Technique  for  Evaluating 
Command  and  Control  Systems 

To  determine  the  criticality  of  various 
problems  that  may  arise  under  operational 
conditions.  Time  to  correct  errors  and 
number  of  times  the  operator  fails  to 
correct  errors  are  determined. 

(13)  System  Load  Mission  Plan 

To  determine  efficiency  of  human  and 
system  performance  under  heavy  load 
conditions  and  to  determine  if  the 
combination  of  operators  and  equipment 
specified  can  successfully  handle  the 
maximum  system  load  specified  by  the 
operational  requirements.  An  extensive 
list  of  the  performance  measures  utilized 
in  each  of  the  operational  analyses  is 
presented  in  this  report. 
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(1*4)  Programmed  System-Performance 
Measuring  Equipment 

To  provide  an  objective  means  of 
evaluating  performance  on  simulators 
and  avionics  equipment.  The  equip¬ 
ment  used  was  the  Digital  System 
Synthesizer,  a  punched  paper  tape 
recorder ,  a  computer  and  necessary 
software . 

(15)  DA COLS 

To  describe  an  automated  method  of 
recording  large  amounts  of  time  and 
event  data  in  a  format  which  can  be 
readily  reduced  and  analyzed  by  com¬ 
puter  processing. 

(16)  Open-Ended  Maintenance  Questionnaire 

To  present  a  questionnaire  designed  to 
collect  information  on  maintenance 
activities  performed  during  the  test 
of  new  systems.  The  questionnaire  is 
of  the  open-ended  type  and  was  compiled 
by  taking  those  questions  judged  to  be 
the  best  from  a  large  number  which  have 
been  previously  used  in  system  evalua¬ 
tion.  Subjective  judgment  was  the  only 
criteria  used  in  selecting  these  ques¬ 
tions  and  the  questionnaire  has  not  been 
valided  in  a  system  test  situation. 
However,  it  is  hoped  that  it  may  prove 
useful  in  reducing  duplication  of  effort. 
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1.  State  of  the  Art  Review 
of  the  Process 

1.1  General  System 
Measurements 

1.2  System  Taxonomy 
Model  (STM) 

1.3  Overall  Conceptual 
Process  Model  (CPM) 

2.  Contextual  Components 
jf  the  Process 

2.1  System  Definition 

2.2  Mission  Definition 

2.3  Environment  Definition 
2. A  General  Constraints 

2.5  Performance 
Requirements,  Ultimate 

2.6  Performance 
Criteria,  Ultimate 


1.1  SAINT  (Systems  Analysis  of  Integrated 
Networks  of  Tasks)  provides  a  graphic 
symbol  set  for  diagramming  event  sequences. 
SAINT  is  a  combination  of  a  set  of  network 
symbols  (modeling  vehicle)  and  simulation 
(analysis  technique).  A  topdown  analysis 
is  employed  such  that  the  system  is  defined 
in  terms  of  missions  with  the  output  of 
SAINT  consisting  of  task  and  mission 
performance  estimates. 

2.2  A  mission  was  defined  as  a  network  of  tasks 
performed  by  a  crew  of  operators  having  a 
complement  of  equipment  in  the  face  of 
environmental  factors. 


3.  Analytic  Components 

of  the  Process  2.3 

3. 1  Practical  Measurable 
Attributes 

3.2  Practical  Attribute 
Measures 

3.3  Performance 
Requirements,  Specific 

3.4  Performance 
Criteria,  Specific 

3.5  Measurement  Procedures  3*1 

4.  Planning  Components 
of  the  Process 

4.1  Analytic  Methods 

4.2  Parameter  Determinations 

4.3  Apparatus  for  Testing 

4.4  Personnel  for  Testing 

4.5  Test  Plans 

5.  Application  Components 
of  the  Process 

5. 1  Test  Execution 

5.2  Data  Analysis 

5.3  Findings  Interpretation 

5.4  Conclusions  and 
Recommendations 


Workload  stress  is  the  time  pressure 
imposed  on  an  operator  by  a  discrepancy 
between  the  amount  of  work  to  be  done  and 
the  time  remaining  for  doing  it.  The 
effects  of  this  stress  are  reflected  in  the 
operator's  task  duration  and  task  success. 

The  set  of  problems  focuses  on  task  alloca¬ 
tion,  operator  workload,  and  environmental 
stressors.  It  is  conjectured  that  network 
concepts  and  symbols  can  be  developed  that 
will  permit  the  modeling  of  one  or  more 
operators  performing  an  assigned  set  of 
tasks  within  the  context  of  a  specific 
mission  and  the  operating  environment  for  a 
mission.  Once  the  network  concepts  and 
symbols  are  designed,  a  simulation  program 
can  be  developed  for  analyzing  mission 
performance  as  a  function  of  operator  and 
environmental  variables. 


6.  Further  Research  Areas 

6.1  Measurement  System 
Limitations 

6.2  Research  Potentials/ 
Priorities 

6.3  Research  Planning 
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6.2  The  following  points  are  proposed: 

(1)  Verification  of  the  factors  and 
relations  included  in  the 
characterization  of  task  performance. 

(2)  Development  of  new  concepts  in  order 
to  model  tasks  that  require  continuous 
monitoring,  queuing  and  resource 
allocation. 

(3)  Extend  the  treatment  of  task  type  and 
the  method  by  which  operators  are 
assigned  to  tasks. 


Wright-Patterson  AFB,  OH:  Air  University,  School  of  Systems  and  Logistics, 
November  1970.  (AD-876  006). 
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1.  State  of  the  Art  Review 
of  the  Process 

1.1  General  System 
Measurements 

1.2  System  Taxonomy 
Model  (STM) 

1.3  Overall  Conceptual 
Process  Model  (CPM) 

2.  Contextual  Components 
of  the  Process 

2.1  System  Definition 

2.2  Mission  Definition 

2.3  Environment  Definition 

2. U  General  Constraints 

2.5  Performance 
Requirements,  Ultimate 

2.6  Performance 
Criteria,  Ultimate 

3.  Analytic  Components 
of  the  Process 

3.1  Practical  Measurable 
Attributes 

3.2  Practical  Attribute 
Measures 

3 . 3  Performance 
Requirements,  Specific 

3 .  *4  Performance 

Criteria,  Specific 
3.5  Measurement  Procedures 

l|.  Planning  Components 
of  the  Process 

44.1  Analytic  Methods 

44.2  Parameter  Determinations 

14.3  Apparatus  for  Testing 
14.44  Personnel  for  Testing 
*1.5  Test  Plans 

3.  Application  Components 
of  the  Process 

5.1  Test  Execution 

5.2  Data  Analysis 

5.3  Findings  Interpretation 
5.*4  Conclusions  and 

Recommendations 

6.  Further  Research  Areas 

6.1  Measurement  System 
Limitations 

6.2  Research  Potentials/ 
Priorities 

6.3  Research  Planning 


The  particular  method  or  combination  of 
methods  employed  in  an  evaluation  must  be 
fitted  to  the  given  situation  in  light  of 
such  variables  as  the  level  of  research  and 
the  stage  of  the  development-cycle  being 
evaluated,  along  with  a  host  of  other 
situational  factors.  By  using  qualitative 
means  to  assist  in  formulating  quantitative 
judgments,  management-decision  makers  can 
predict,  measure,  and  evaluate  research  and 
development  efforts  and  results. 

Methods  of  evaluating  research  potential, 
performance,  and  effectiveness  may  be 
quantitative,  qualitative,  or  combinations 
of  both.  Quantitative  methods  employ 
mathematical  equations  or  models  that  are 
coupled  with  dollar  evaluation  to  arrive  at 
a  figure  of  the  research's  effectiveness. 

Quantitative  methods  evaluate  the  research 
via  the  profits  created  and  the  improved  or 
cost-reduced  products,  techniques,  proc¬ 
esses,  and  materials.  Qualitative  evalu¬ 
ation  is  subjective  judgment,  that  is,  the 
composite  judgment  of  qualified  and  respon¬ 
sible  management  and  research  personnel 
arrived  at  through  logical,  not  mathemati¬ 
cal,  procedures  and  devices.  Qualitative 
methods  are  applied  to  the  quality  of 
research  results,  the  degree  of  research 
efficiency,  the  results  of  long-range 
research,  and  the  intangible  products  of 
research  such  as  publications. 
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1.2  A  basic  criterion  for  categorizing  the 

level  of  research  effort  is  the  degree  of 
uncertainty  of  research  results.  The  four 
classifictions  of  research  are  pure,  basic, 
applied,  and  developmental  with  pure 
research  at  one  end  of  the  spectrum  of 
uncertainty  and  developmental  research  at 
the  other.  The  principal  implication  of 
the  spectrum  of  uncertainty  on  the  evalu¬ 
ation  of  research  effort  is  that  the  result 
of  research  at  the  less  uncertain  end  of 
the  spectrum  may  be  predicted  and  measured 
with  some  degree  of  confidence  whereas 
those  at  the  upper  end  are  more  difficult 
to  predict  and  measure. 
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1.  State  of  the  Art  Review 
of  the  Process 

1.1  General  System 
Measurements 

1.2  System  Taxonomy 
Model  (STM) 

1.3  Overall  Conceptual 
Process  Model  (CPM) 


2.  Contextual  Components 
of  the  Process 

2.1  System  Definition 

2.2  Mission  Definition 

2.3  Environment  Definition 

2.4  General  Constraints 

2.5  Performance 
Requirements,  Ultimate 

2.6  Performance 
Criteria,  Ultimate 


3. 


Analytic  Components 
of  the  Process 
3.1  Practical  Measurable 
Attributes 

3-2  Practical  Attribute 
Measures 
3-3  Performance 

Requirements,  Specific 

3.4  Performance 
Criteria,  Specific 

3.5  Measurement  Procedures 


1.1  Like  most  statistical  methods,  multidimen¬ 
sional  scaling  is  merely  a  procedure  for 
converting  the  numerical  data  resulting 
from  an  experiment  into  a  more  interpret¬ 
able  form.  As  with  most  statistical 
methods,  it  is  the  hope  of  the  experimenter 
that  not  too  much  of  the  information  availa¬ 
ble  in  the  sample  will  be  lost  during  the 
course  of  analyses.  The  problem  of  mini¬ 
mizing  the  loss  of  information,  therefore, 
is  really  two  problems  -  one  of  estimation 
and  the  other  of  a  scaling  procedure. 

The  second  problem  has  received  consider¬ 
able  attention,  while  the  first  has 
received  virtually  none.  The  purpose  of 
this  study  is  to  show  the  importance  of  the 
estimation  problem  and  to  make  some 
appropriate  comments. 


4.  Planning  Components 
of  the  Process 

4.1  Analytic  Methods 

4.2  Parameter  Determinations 

4.3  Apparatus  for  Testing 

4.4  Personnel  for  Testing 

4.5  Test  Plans 

5-  Application  Components 
of  the  Process 

5.1  Test  Execution 

5.2  Data  Analysis 

5.3  Findings  Interpretation 

5.4  Conclusions  and 
Recommendations 

6.  Further  Research  Areas 

6.1  Measurement  System 
Limitations 

6.2  Research  Potentials/ 
Priorities 

6.3  Research  Planning 
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1 .  State  of  the  Art  Review  1.1 

of  the  Process 

1.1  General  System 
Measurements 

1.2  System  Taxonomy 
Model  (STM) 

1.3  Overall  Conceptual 
Process  Model  (CPM) 

2.  Contextual  Components 

of  the  Process  c 

2.1  System  Definition 

2.2  Mission  Definition 

2.3  Environment  Definition 

2.4  General  Constraints 

2.5  Performance 
Requirements,  Ultimate 

2.6  Performance 
Criteria,  Ultimate 

3.  Analytic  Components 
of  the  Process 

3.1  Practical  Measurable 
Attributes 

3.2  Practical  Attribute 
Measures 

3.3  Performance 
Requirements,  Specific 

3.4  Performance 
Criteria,  Specific 

3.5  Measurement  Procedures 

4.  Planning  Components 
of  the  Process 

4.1  Analytic  Methods 

4.2  Parameter  Determinations 

4.3  Apparatus  for  Testing 

4.4  Personnel  for  Testing 

4.5  Test  Plans 

5.  Application  Components 
of  the  Process 

5.1  Test  Execution 

5.2  Data  Analysis 

5.3  Findings  Interpretation 

5.4  Conclusions  and 
Recommendations 

6.  Further  Research  Areas 

6.1  Measurement  System 
Limitations 

6.2  Research  Potentials/ 

Priorities 

6.3  Research  Planning 


The  purpose  of  this  study  is  to  determine 
the  use  of  fault  tree  analysis  in  operation¬ 
al  test  planning  for  military  systems.  The 
research  attempted  to  demonstrate  how  the 
logic  of  fault  trees  could  be  used  in  an 
area  other  than  safety  and  reliability.  In 
this  case  it  was  in  test  design. 

A  detailed  discussion  was  given  as  to  the 
procedures  used  in  fault  tree  analysis: 
fault  tree  terminology  and  symbology, 
system  definition,  fault  tree  construction 
and  qualitative  analysis,  the  identifica¬ 
tion  and  ranking  of  components  and  subsys¬ 
tems  through  fault  tree  analysis  permits 
direction  of  tests  towards  the  weak  links 
in  the  system.  System  weakness  and  the 
causes  of  important  failures  can  be 
explored  in  testing.  Fault  tree  analysis 
increases  system  understanding.  It  pro¬ 
vides  a  vehicle  to  explore  system  alterna¬ 
tives.  Structural  analysis  frees  the 
method  from  reliability  data  which  may  not 
be  available  due  to  the  nature  of  the 
system  or  the  equipment  state  of  the  art. 

Fault  tree  analysis  is  a  binary  modeling 
scheme  which  ignores  partial  failures.  It 
is  also  situation  specific,  that  is,  each 
tree  is  constructed  only  about  the  failure 
of  interest.  Hence,  it  can  explore  only 
one  critical  issue  for  each  model 
developed. 

An  example  was  directed  toward  the  opera¬ 
tional  testing  of  automated  command  and 
control  systems  for  use  in  the  U.S.  Army 
division. 
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1.  State  of  the  Art  Review 
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Model  (STO) 

1.3  Overall  Conceptual 
Process  Model  (CPM) 

2.  Contextual  Components 
of  the  Process 

2.1  System  Definition 

2.2  Mission  Definition 

2.3  Environment  Definition 

2.4  General  Constraints 

2.5  Performance 
Requirements,  Ultimate 

2.6  Performance 
Criteria,  Ultimate 

3.  Analytic  Components 
of  the  Process 

3.1  Practical  Measurable 
Attributes 

3.2  Practical  Attribute 
Measures 

3.3  Performance 
Requirements,  Specific 

3.4  Performance 
Criteria,  Specific 

3-5  Measurement  Procedures 

4.  Planning  Components 
of  the  Process 

4.1  Analytic  Methods 

4.2  Parameter  Determinations 

4.3  Apparatus  for  Testing 

4.4  Personnel  for  Testing 

4.5  Test  Plans 

5.  Application  Components 
of  the  Process 

5.1  Test  Execution 

5.2  Data  Analysis 

5.3  Findings  Interpretation 

5.4  Conclusions  and 
Recommendations 

6.  Further  Research  Aruas 

6.1  Measur**«nt  System 
Limitations 

6.2  Research  Potentials/ 
Priorities 

6.3  Research  Planning 


This  report  presents  a  model  of  piloted 
weapon  delivery  through  the  application  of 
mathematical  models  of  the  hixnan  operator's 
performance  characteristics  to  the  dynamic 
description  of  the  combined  control-dis¬ 
play-vehicle  system.  By  doing  this  an 
understanding  of  the  interaction  and  rela¬ 
tive  importance  of  the  various  elements  of 
the  system  were  obtained. 

A  statistical  model  of  the  propagation  of 
these  pilot-induced  errors  into  impact 
error  is  then  developed  by  considering  each 
of  the  pilot  inputs  to  be  a  random 
variable.  A  method  for  including  the 
effect  of  pilot  compensation  for  an 
observed  error  in  one  of  the  variables  with 
an  intentional  deviation  in  another  is  also 
introduced.  An  analytical  model  of  the 
human  pilot  is  used  to  estimate  the 
tracking  error  from  the  controlled-element 
dynamics  and  the  turbulence  environment. 
Typical  pilot  performance  in  controlling 
the  other  task  variables  is  estimated 
empirically  from  piloted  simulation 
results.  Once  the  system  is  modeled  in 
this  manner,  the  effect  of  display, 
computational,  and  dynamic  changes  on 
weapon  delivery  accuracy  can  be  readily 
determined.  A  reiterative  design  approach 
is  used  to  improve  the  system  using  impact 
point  accuracy  as  a  figure  of  merit;  thus, 
the  effects  of  changes  in  displayed 
information,  of  computational  aids  provided 
to  the  pilot,  and  of  varying  degrees  of 
control  system  augmentation  can  all  be 
compared  on  a  common  scale. 
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The  approach  is  applied  to  a  gun-firing 
pass  of  a  typical  tactical  fighter  aircraft 
equipped  with  a  conventional  fixed  optical 
sight.  A  sequence  of  design  improvements 
is  proposed  which  are  predicted  analytical¬ 
ly  to  improve  impact  accuracy.  The  pro¬ 
posed  changes  are  then  tested  in  a  com¬ 
plete,  six-degree-of-freedom,  piloted  simu¬ 
lation  in  order  to  validate  the  modeling 
technique  and  determine  the  extent  to  which 
predicted  improvements  materialize.  The 
simulation  discloses  that  tracking  error  is 
the  principal  contributor  to  strafing 
inaccuracy  and  that  poor  tracking  capabili¬ 
ty  in  either  the  longitudinal  or  lateral 
axis  will  degrade  the  accuracy  in  both 
axes.  A  five-fold  increase  in  single 
shot/pass  hit  probability  is  demonstrated 
which  can  be  primarily  attributed  to  a  55 % 
reduction  in  lateral  tracking  error.  The 
mathematical  model  of  the  air-to-ground 
gun-firing  pass  is  revised  to  incorporate 
this  coupling  effect  and  other  effects 
noted  in  the  piloted  simulation. 
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1.  St»te  of  the  Art  Review 
of  the  Process 

1 . 1  General  System 
Measurements 

1.2  System  Taxonomy 
Model  (STM) 

1.3  Overall  Conceptual 
Process  Model  (CPM) 


1.3  Generally  speaking,  this  document  describes 
the  trade-offs  made  during  the  various 
phases  of  acquisition  (e.g.,  of  naval 
vessels),  explains  TPM  (Technical 
Performance  Measurement),  and  provides 
guidelines  for  using  the  TPM  outputs  in 
trade-off  analyses. 


2.  Contextual  Components 

of  the  Process  2.  1 

2.1  System  Definition 

2.2  Mission  Definition 

2.3  Environment  Definition 

2.4  General  Constraints 

2.5  Performance 
Requirements,  Ultimate 

2.6  Performance 
Criteria,  Ultimate 

3.  Analytic  Components 
of  the  Process 

3.1  Practical  Measurable 
Attri butes 

3.2  Practical  Attribute  2.2 

Measures 

3.3  Performance 
Requirements,  Specific 

3.4  Performance 
Criteria,  Specific 

3-5  Measurement  Procedures 

4.  Planning  Components 

of  the  Process  a  . 

4.1  Analytic  Methods 

4.2  Parameter  Determinations 

4.3  Apparatus  for  Testing 

4.4  Personnel  for  Testing 

4.5  Test  Plans 

5.  Application  Components 
of  the  Process 

5.1  Test  Execution 

5.2  Data  Analysis 

5.3  Findings  Interpretation 

5.4  Conclusions  and 
Recommendations 


The  purpose  of  this  system  for  making 
trade-offs  was  to  specify  those  performance 
measurement  factors  relevant  to  ship 
acquisition  and  to  specify  standards  for 
using  TPM  outputs  to  make  the  necessary 
trade-offs  for  ship  design  decisions.  The 
system  allowed  for  the  determination  of 
which  trade-offs  were  possible  during  each 
phase  of  ship  acquisition  and  gave  a 
"user’s  guide"  for  the  process  of  TPM. 

The  primary  goal  of  the  trade-off  TPM 
system  was  the  specification  of  the 
performance/schedule/cost  parameters  that 
eventually  determine  the  system's 
effectiveness  and/or  the  success 
probability  of  the  mission. 

The  trade-off  problem  was  attacked  througn 
several  stages  of  analysis: 

(1)  The  Validation  Stage,  or  System 
Contract  Design  (i.e.,  do  individual 
systems  fulfill  requirements?) 

(2)  Detail  Design  and  Construction: 
factors  such  as  cost  and  delay,  legal 
approval,  ongoing  modification,  etc., 


6.  Further  Research  Areas 

6.1  Measurement  System 
Limitations 

6.2  Research  Potentials/ 
Priorities 

6.3  Research  Planning 
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must  be  considered  during  the 
blueprinting  of  the  ship  design. 

(3)  Deployment  and  Disposal:  technological 
and  "threat"  changes  should  be 
considered  in  an  evaluation  of  the 
ship's  optimum  capability;  the  ship 
itself,  as  a  total  system,  has  a  much 
longer  life  than  its  subsystems. 

3.3  Specifically,  TPM  was  used  to  predict  the 
degree  of  actual  or  potential  achievement 
of  certain  technical  objectives  of  a  system 
or  subsystem,  and  causally  analyzes  the 
variance  between  achievement  and  goals. 

3.4  To  quantify  technical  performance,  the 
following  steps  must  be  taken: 

(1)  Identify  performance  variables 
(inputs)  crucial  for  success  and 
relate  them,  by  means  of  equations,  to 
design  variables  or  outputs. 

(2)  Through  questioning  of  design 
personnel,  subjective  probability 
distributions  can  be  made  for  the 
design  variables. 

(3)  To  estimate  the  probability  of 
obtaining  desired  performance  or 
meeting  certain  technical  objectives, 
appropriate  techniques,  e.g., 
simulation,  can  be  used. 

(1J)  Changes  in  the  probability  of 

achieving  target  goals  in  performance 
should  be  monitored. 

1J.5  In  order  to  implement  the  TPM  program,  the 
following  elements  play  a  determining  role: 
Parameter  Selection  and  Documentation  of 
Detail;  Construction  of  TPM  models; 
Profiling  Parameters;  Planning  the  TPM; 
Assessing  Organizational  Participation; 
Preparation  of  Reports  and  Formats;  Data 
Analysis  and  Predictions. 
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State  of  the  Art  Review 
of  the  Process 

1.1  General  System 
Measurements 

1.2  System  Taxonomy 
Model  (STM) 

1.3  Overall  Conceptual 
Process  Model  (CPM) 


1.3  This  document  presents  a  summary  of 
measures  of  effectiveness  used  by  the 
Navy’s  Operational  Test  and  Evaluation 
Force  (OPTEVFOR). 

2.1  Several  platforms,  systems  and  subsystems 
were  considered. 


2.  Contextual  Components 

of  the  Process  2.2 

2.1  System  Definition 

2.2  Mission  Definition 

2.3  Environment  Definition 

2. H  General  Constraints 

2.5  Performance  2.1 

Requirements,  Ultimate 
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<1.5  Test  Plans 


The  minimal  operational  situation  was 
described  for  each  system  or  subsystem 
discussed. 

A  reference  source  was  provided  based  on  a 
review  of  OPTEVFOR  reports,  of  measures  of 
effectiveness  used  in  Naval  warfare  and 
previous  OPTEVFOR  projects. 

The  specific  criterion  performance 
requirements  were  presented  followed  by 
specific  measures  of  effectiveness  for  each 
system. 

The  scope  of  this  handbook  is  limited  to 
effectiveness  measures  only.  Material 
reliability  and  human  factors  measures  are 
not  included.  The  handbook  is  for  guidance 
only  and  the  coverage  is  not  intended  to  be 
complete. 


5.  Application  Components 
of  the  Process 

5.1  Test  Execution 

5.2  Data  Analysis 

5.3  Findings  Interpretation 
5. Conclusions  and 

Recommendations 


6.  Further  Research  Areas 

6.1  Measurement  System 
Limitations 

6.2  Research  Potentials/ 
Priorities 
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5.1  Test  Execution 
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2.1  The  Defense  Satellite  Communications  System 
Phase  III  (DSCS  III)  consisted  of  four 
operational  satellites  that  were 
synchronized  with  the  earth’s  movements, 
and  two  spare  satellites,  positioned 
equidistant  at  equal  intervals  from  the 
earth’s  equator.  Several  types  of  "Earth 
Terminals”  (ETs)  als.i  comprised  the  system 
at  various  locations  around  the  world,  with 
four  to  eight  SCCE  facilities  (Satellite 
Configuration  Control  Elements). 

2.2  The  primary  "mission"  of  the  DSCS  was  an 
increased  communications  capability, 
particularly  an  improved  ability  to  operate 
in  an  Electronic  Warfare  (EW)  environment. 

2.5  The  ultimate  performance  requirement  of  the 
test  design  plan  was  the  collection  and 
analysis  of  data  regarding  operational 
effectiveness,  vulnerability  of  the  system 
in  an  EW  environment,  assessment  of  the 
operational,  maintenance,  and  logistic 
support  concepts  of  the  system,  RAM  data 
(reliability,  availability,  and 
maintainability) . 

3.1  Operational  issues  included  the  areas  of 
mission  performance,  vulnerability, 
training,  organization,  RAM,  and 
safety/human  factors  engineering. 

3.2  These  will  be  measured  through  such  indices 
as  Bit  Error  Rate  (BER),  Test  Tone-to-Noise 
Ratio  (TTNR),  and  carrier-power-to-noise 
spectral  density  ratio.  These  measurements 


Further  Research  Areas 

6.1  Measurement  Systea 
Limitations 

6.2  Research  Potentials/ 
Priorities 

6.3  Research  Planning 


A-361 


Topics  Relevant 
to  System  Development 
and  Evaluation  Technolo 


-Vi 


iTopic! 
I  No.  I 


ABSTRACT 


are  used  to  determine  if  a  link  (circuits) 
will  pass  data  traffic. 

Test  factors  were  identified  which  would 
probably  influence  the  operational 
effectiveness  of  DSCS  III;  examples  of 
those  factors  which  are  varied  included 
communications  access,  user  community, 
antenna  coverage,  satellite  configuration, 
etc. 


There  are  four  test  conditions  which 
required  action  from  the  satellite  control 
system:  a  "normal"  mode,  "degraded"  mode, 
"hostile"  electronic  warfare  environmental 
mode,  and  subnetwork  control  mode. 

General  data  collection  involved 
information  on  performance  of  the  system 
with  regard  to: 

(1)  Reception  of  telecommunications 
service  order  from  Defense 
Communication  Agency  Operation  Center. 

(2)  Performance  of  network  analysis  using 
resource  allocation  software  in 
network  computer. 

(3)  Instruction  of  satellite  controller  to 
command  necessary  communications 
payload  change. 

(4)  Instruction  of  earth  terminal 
operators  to  establish  link  with 
parameters  as  specified. 
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approach  in  evaluation  of  parameter  changes  on  the  human  operator  under 
thermal  stress.  Ergonomics ,  1978,  21(1),  35-48. 
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_ ABSTRACT _ 

A  feature  selection  approach  to  define 
changes  in  parameters  of  the  human  operator 
is  used  in  this  study. 

The  human  operator  was  involved  in  a 
closed-loop  tracking  task  simulating  a 
flight  in  a  very  unstable  aircraft  or  a 
very  stable  aircraft. 

The  experiment  was  conducted  in  a 
laboratory  situation  under  thermal  stress 
conditions.  The  subjects  were  exposed  to 
an  environment  of  50°  C  dry  bulb  and  30°  C 
wet  bulb  to  simulate  a  pilot  forced  to  sit 
prior  to  take  off  in  a  heat-soaked 
aircraft. 

By  choosing  a  class  of  features  which 
characterize  the  general  shape  of  the  human 
operator's  transfer  function,  significant 
changes  in  these  characteristics  were  found 
to  occur  under  thermal  stress  conditions. 

An  F-Ratio  Test  was  used  with  an  analysis 
of  variance  to  test  the  significance  of  the 
change  in  those  features  which  describe  the 
input/output  characteristics  of  the  human. 

The  human's  parameters  are  assumed  to  be 
characterized  by  second-order  transfer 
functions,  and  features  of  the  transfer 
function  are  chosen  which  describe  its 
general  shape. 

Four  healthy  and  highly  trained  male 
subjects  were  used  in  this  experiment. 
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4.5  These  subjects  were  exposed  for  1  hour  to  a 
simulated  heat-soaked  aircraft  environment. 
They  performed  a  single  dimension  compen¬ 
satory  tracking  task  for  5  minutes 
duration,  separated  by  5  minutes  of  rest. 
The  tasks  represented  flying  a  very 
unstable  aircraft  or  a  very  stable  aircraft 
under  vertical  wind  buffet. 

Each  subject  participated  in  six  experi¬ 
mental  conditions,  three  control  runs  and 
three  exposures  to  the  heat-loading 
environment.  During  the  experiment  the 
subject  maintained  one  of  three  conditions 
of  water-electrolyte  balance.  He  either 
drank  nothing  or  replaced  weight  losses 
with  water  or  NaCl  solution.  The  subject 
urinated  and  blood  samples  were  drawn 
periodically.  Mean  skin  temperature, 
rectal  temperature,  weight  los3,  heart 
rate,  air  temperature,  water  temperatures, 
and  humidity  were  recorded  along  with 
tracking  performance  parameters. 

5.3  The  results  of  the  experiment  indicated 

that  the  effects  of  thermal  stress  (and  the 
type  of  electrolyte  used  in  the  recovery 
period)  will  significantly  change  the 
input/output  characteristics  of  the  human 
operator . 
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This  report  described  an  in-flight 
evaluation  of  four  cockpit-controller 
configurations  in  a  variable  stability 
airplane  with  pilot. 

Evaluation  was  based  on  three  tasks:  up 
and  away  cruise  condition  maneuvers,  low 
level  terrain  following  simulated  altitude, 
approach  landing,  and  take-off. 

Each  task  was  performed  under  conditions  of 
different  simulated  static  and  dynamic 
characteristics  of  a  B-1  type  airplane. 

Qualitative  and  quantitative  data  were 
obtained  consisting  of  pilot  inflight 
comments  and  pilot  ratings  of  handling 
characteristics  and  tracking  error. 

The  pilot  inflight  comments  were  responses 
to  a  prepared  comment  guide  and  the  last 
Cooper-Harper  handling  rating  scale  was 
utilized 

Much  use  of  simple  average  numerical  rating 
data  were  used  in  pilot  rating  analysis; 
average  of  replication,  average  of  pilots, 
etc.  In  the  tracking  tasks,  raw  data  were 
oscillograph  recorded.  A  limited  amount  of 
raw  data  were  reduced  to  a  variance 
spectral  density  form.  This  form  indicates 
variance  as  a  function  of  frequency  and 
shows  the  variance  present  in  any  frequency 
bandwidth  throughout  the  spectrum  (and 
hence  a  frequency  bandwidth  for  maximum 
variance)  and  the  overall  variance  for  the 
paraaenters  of  the  maneuvers. 
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A  conventional  wheel  and  column  and  three 
new  concepts:  a  circumferential  wheel  and 
colinm;  a  circumferential  wheel  and  column 
with  hand  controller;  and  a  duel  side  arm 
configuration  were  evaluated  using  a  CAL 
B-26  variable  stability  airplane  of  the  B-1 
type. 

Four  pilots  -  -  two  USAF  and  two  CAL 
participated  in  these  tests. 

The  pilots  flew  all  four  controller  configu 
rations  four  times  each  through  each  of  the 
three  tasks. 


Results  of  the  analysis  indicated  that  all 
three  new  concepts  would  be  accepted  by 
pilots  of  large  airplanes  with  only  a 
nominal  associated  learning  period  and  that 
all  three  new  concepts  are  preferable  to 
the  conventional  equipment.  Despite  some 
detailed  design  deficiencies,  the  dual  side 
arm  configuration  was  most  preferred. 
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The  Sandia  Human  Error  Rate  Bank  (SHERB)  is 
an  evolving  file  of  Human  Error  Rates  (HER) 
on  specific  tasks  associated  with  component 
assembly,  equipment  installation,  system 
operation,  and  maintenance.  It  is  intended 
to  serve  as  a  data  base  to  allow  analysts 
to  estimate  the  probability  of  human  error 
and  its  impact  on  system  performance. 

If  human  error  is  defined  as  any  variant  of 
human  performance  that  reduces  the 
probability  of  system  or  mission  success, 
then  failures  due  to  human  error  can  be 
treated  very  similarly  to  component 
failures,  i.e.,  human  errors  can  be 
predicted  as  a  probabilistic  function  of 
the  variables  determining  or  influencing 
that  human  performance  related  to  system 
performance. 

Development  of  an  accurate  data  base  of 
human  error  rates  is  impeded  by  the 
following  factors:  (1)  Accidents  and 
mission  failures  resulting  from  human  error 
are  not  reported  with  the  same  regularity 
and  accuracy  as  equipment  failures;  there 
is  a  lack  of  good  feedback  data.  (2)  The 
data  that  are  available  vary  widely  in 
terminology,  manner  of  development,  and 
level  of  reporting;  there  is  a  lack  of 
standardization. 
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1.1  The  methodology  of  estimation  and  control 
theory  is  considered  in  terms  of  response, 
stability,  estimation,  and  control  of 
linear  dynamic  systems.  Within  the  context 
of  discrete  time  systems,  multi-input, 
multi-output,  nth-order  linear  systems  are 
discussed,  and  general  results  for  optimal 
estimation,  optimal  control,  and  other 
topics  are  presented.  The  application  of 
these  results  to  modeling  human  behavior  is 
considered  with  special  emphasis  on 
man-machine  system  models. 
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2.2  The  primary  goal  of  this  study  was  to 
assess,  through  quantification,  the 
relationships  between  human  resource 
factors  and  nuclear  system  safety,  and  to 
use  these  data  to  simulate  nuclear  system 
maintainance  operations. 
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3.1  Thirteen  factors  considered  relevant  to 
nuclear  safety  maintenance  operations  were 
identified:  motivation,  group  cohesive¬ 
ness,  behavioral/emotional  stability, 
fatigue,  leadership,  organizational 
structure,  task  complexity,  written 
manuals,  job  skill  proficiency  aptitude, 
training,  work  experience,  and  environ¬ 
mental  conditions. 

3.2  Each  of  these  thirteen  factors  were 
quantified  for  computer  simulation  use; 
measures  of  quantification  included 
psychological  tests,  supervisor  ratings, 
and  biographic  data.  These  were 
consolidated  into  one  questionnaire  and 
administered  to  each  missile  technician. 
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Data  were  analyzed  through  a  computer 
simulation  model.  The  Short-Range  Attack 
Missile  (SRAM)  network  was  fed  into  the 
computer  with  each  task  assigned  four 
descriptors:  time,  crew  type,  hazard 
value,  and  maintenance  task  category. 
Equations  describing  functions  between  the 
h"man  resource  factors,  time,  and  hazard 
were  correlated  by  the  System  Analysis  of 
Integrated  Networks  of  Tasks  (SAINT) 
computer  simulation  model  and  cycled  about 
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500  times  to  approximate  how  a  crew 
possessing  given  characteristics  of 
motivation,  work  experience,  fatigue,  and 
training  under  given  environmental 
conditions  would  perform. 

Test  parameters  were  described  by  the 
maintenance  and  ground  handling  tasks 
involving  the  SRAM  system.  Tasks  were 
described  by  time  required  for  completion, 
career  field  involved  in  the  task,  degree 
of  hazard  inherent  in  the  task,  and  task 
classification  by  either  transport, 
checkout,  or  assemble/disassemble. 

Results  of  supervisor  rankings  of  techni¬ 
cians  indicated  that  those  techniciars  who 
had  more  experience,  higher  aptitude 
levels,  were  more  stable  emotionally,  had 
less  fatigue,  and  greater  morale  were  more 
satisfied  with  their  jobs.  The  above 
factors  were  also  positively  correlated 
with  accuracy  and  task  performance. 


> 


Schaeffer,  K.H.,  Fink,  J.B.,  Rappaport,  M.,  Wainstein,  L.  4  Erickson,  C.J.  The 
knowledgeable  analyst:  An  approach  to  structuring  man-machine  systems  (SRI 
No.  IMU-3546).  Menlo  Park,  CA:  Stanford  Research  Institute,  February  1963. 
(AD-297  «32). 
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The  objectives  of  systems  analysis  are  to 
analyze  specific  systems  in  order  to  solve 
predesignated  problems.  These  objectives 
are  at  variance  with  those  of  the  empirical 
sciences,  which  attempt  to  discover  general 
laws.  Since  the  two  objectives  differ,  the 
method  of  systems  analysis  differs  from  the 
method  of  science.  In  an  attempt  to  evolve 
a  general  method  for  systems  analysis,  the 
matrix-network  approach  for  the  analysis  of 
complex  man-machine  systems  is  presented. 
This  approach  consists  of  seven  steps  which 
show  how  a  system  can  be  structured  and  how 
mathematical  models  of  systems  aspects  can 
be  incorporated  into  the  over-all  analysis. 
However,  some  of  these  steps  involve, 
besides  formal  rules,  the  Judgment  of  a 
knowledgeable  analyst.  To  delve  deeper 
into  this  judgment  function,  various 
logical,  methodological,  and  psychological 
aspects  concerning  this  function  are 
discussed  by  different  authors.  On  the 
basis  of  these  discussions  the  principal 
author  develops  requirements  which  must  be 
met  by  successful  approaches  to  the 
structuring  of  complex  systems. 
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In  a  reconnaissance  strike  or  reconnais¬ 
sance/strike  system  there  is  a  complex  of 
equipment  and  men.  One  would  assume  that 
maximum  effectiveness  of  the  man-machine 
system  would  be  obtained  by  utilizing  the 
best  men  and  the  best  machines;  fcnis 
reasoning,  however,  conflicts  with  problems 
of  expenses,  unavailability,  and  the  inabil¬ 
ity  to  determine  the  best  men  for  the  job. 

Characteristics  of  the  men,  the  equipment, 
and  the  mission  interact  such  that 
variations  in  equipment  and/or  situation 
will  have  an  effect  on  the  men  in  the 
system.  In  other  words,  a  system  approach 
is  essential  when  dealing  with  a  system. 

The  performance  measures  in  this  study  were 
percent  detection,  accuracy,  and  reaction 
time  or  slant  range  at  detection. 

The  ideal  observer  would  detect  all  of  the 
targets  (%D  =  100}),  mistake  no  targets 
(accuracy  =  100%),  and  would  detect  and 
recognize  targets  the  instant  that  their 
images  appeared  on  the  display  (reaction 
time  =  0  seconds) . 

Mission-relevant  performance  criteria 
Included  the  following: 
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The  number  and  percentage  of  targets 
that  are  detected.  An  ideal  observer 
would  detect  all  targets  that  were 
displayed  with  some  minimum  image 
quality. 
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(2)  An  ideal  observer  would  make  no 
mistakes  in  designating  targets  as 
such. 

(3)  Targets  should  be  adequately 
recognizable  at  long  slant  range. 


Shapero,  A.  &  Bates,  C. ,  Jr.  A  method  for  performing  human  engineering 
analysis  of  weapon  systems  (WADC  Tech.  Rep.  59-784).  Wright-Patterson  AFB 
OH:  Wright  Air  Development  Center,  Aerospace  Medical  Laboratory,  September 
1959. 
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1.3  A  weapon  System  Analysis  and  Integration 
Model  (SAIM)  has  been  developed  that  in¬ 
cludes  the  system's  human  elements  and 
that  can  be  employed  as  an  aid  in  the 
analysis,  synthesis,  evaluation,  planning, 
and  management  control  of  weapon  systems. 
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A  diagrammatic  matrix  is  used  to  structure 
the  model.  The  system  components  (subsys¬ 
tems)  are  represented  in  the  same  way  as 
the  system.  The  matrix  can  be  described 
as  a  two-dimensional  square.  The  headings 
of  the  rows  and  columns  are  symmetrical 
and  consist  of  the  system  elements.  The 
orientation  of  the  matrix  is  such  that 
the  elements  appearing  as  headings  of  rows 
are  considered  as  affecting  those  appearing 
as  headings  of  columns.  The  matrix  includes 
system  determinants,  system  and  subsystem 
components,  and  system  integration. 

SAIM  is  designed  to  indicate  interactions 
and  to  provide  a  means  for  describing  the 
interactions  at  a  given  time.  The  effects 
of  developmental  changes  that  occur  can  be 
taken  into  account  by  updating  the  matrix. 

The  SAIM  is  a  descriptive  matrix  model  that 
classifies  the  elements  of  a  weapon  system 
into  those  determining  the  nature  and  form 
of  the  system,  those  comprising  the  parts 
of  the  system,  and  those  integrating  the 
parts  of  the  system.  The  approach  used  in 
SAIM  predicates  that  any  system  and  any  of 
its  subsystems  have  the  same  kinds  of  con¬ 
ceptual  elements  and  SAIM  employs  a  scheme 
for  classifying  these  elements  that  is 
applicable  to  any  system  or  any  level. 
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In  SAIM,  system  components  are  described 
in  terms  of  mechanism  modules,  human 
operator  modules  and  facility  modules. 

This  modular  treatment  affords  flexibility 
since  it  allows  each  subsystem  to  be 
handled  in  a  building-block  fashion,  thus 
making  it  easy  to  describe  complex  func¬ 
tions  that  combine  many  items,  and  to 
extend  or  to  modify  the  model  as  the  system 
itself  is  modified  or  extended.  System 
integrators  are  the  elements  that  link  the 
system’s  components  in  a  ’’nonhardware”  or 
abstract  way. 

2.2  In  this  model,  the  mission  statement  is 

always  framed  in  general  terms,  enabling  it 
to  remain  unchanged  while  permitting  the 
incorporation  of  changes  in  performance 
requirements  as  imposed  by  technical,  budget 
or  other  changes  in  the  weapon  system. 

2.*i  Limits  on  the  system  are  imposed  by  the 
state-of-the-art,  by  nature,  and  by  the 
agencies  that  have  cognizance  over  the 
system.  The  constraints  imposed  by  nature 
or  society  relate  to  the  physical  environ¬ 
ment  and/or  human  and  material  resources. 
Constraints  imposed  by  cognizant  agencies 
include  those  that  relate  to  funds  and 
developmental  time. 

2.5  The  performance  requirements  detail  the 

system’s  mission  into  a  set  of  goals,  stan¬ 
dards  and  objectives.  Performance  require¬ 
ments  are  usually  categorized  into  operational 
and  support  requirements.  Operational  re¬ 
quirements  might  include  such  factors  as  kill 
probability,  range,  speed,  etc.  Support 
requirements  could  include  readiness,  main¬ 
tenance  and  servicing,  and  handling. 


A-375 


Sheldon,  H.S.  &  Zagorsk! ,  H.J.  Han-machine  system  evaluation 
Operations  Reporting  Method.  In  R.C.  Blanchard  &  D.H.  Harris 
Han-machine  effectiveness  analysis.  Papers  presented  at  The 
Symposium,  University  of  California  at  Los  Angeles,  June  15, 
(AD-735  718). 


-  The  Normative 
(Eds.), 

Human  Factors 
1967. 


Topics  Relevant  I  i 

System  Development  iTopic! 

I  Evaluation  Technology  {  No.  I _ ABSTRACT 


1.  State  of  the  Art  Review  1.1 

of  the  Process 

1.1  General  System 
Measurements 

1.2  System  Taxonomy 
Model  (STM) 

1.3  Overall  Conceptual 
Process  Model  (CPM) 

2.  Contextual  Components 
of  the  Process 

2.1  System  Definition 

2.2  Mission  Definition 

2.3  Environment  Definition 

2.4  General  Constraints 

2.5  Performance 
Requirements,  Ultimate 

2.6  Performance 
Criteria,  Ultimate 

1.3 

3.  Analytic  Components 
of  the  Process 

3.1  Practical  Measurable 
Attributes 

3.2  Practical  Attribute 
Measures 

3.3  Performance 
Requirements,  Specific 

3.4  Performance 
Criteria,  Specific 

3.5  Measurement  Procedures 

4.  Planning  Components 

of  the  Process  P  « 

4.1  Analytic  Methods  * 

4.2  Parameter  Determinations 

4.3  Apparatus  for  Testing 

4.4  Personnel  for  Testing 

4.5  Test  Plans 

5-  Application  Components 
of  the  Process 

5.1  Test  Execution 

5.2  Data  Analysis 

5.3  Findings  Interpretation 

5.4  Conclusions  and 
Recoomendatl ons 

6.  Further  Research  Areas 

6.1  Measurement  System 
Limitations 

6.2  Research  Potentials/ 

Priorities 

6 . 3  Research  Planning 


It  is  increasingly  evident  that  man-machine 
system  evaluation  needs  techniques  that  are 
radically  different  from  traditional 
methods.  The  authors  call  this  work  area 
systemetrics.  The  overall  purpose  of  the 
systemetric  model  is  to  develop  a 
methodology  permitting  evaluation  of 
man-machine  performance  to  be  based  on  a 
series  of  flexible  standards  reflecting  the 
difficulty  of  the  mission,  in  direct 
contradistinction  to  the  absolute  standards 
approach.  The  systemetric  approach 
requires  intimate  familiarity  with  the 
system. 

One  example  of  the  kind  of  work  that  can  be 
done  is  the  Normative  Operations  Reporting 
Method  (NORM).  This  method  includes  no 
exotic  breakthrough;  its  creativity  lies  in 
its  unique  combination  and  application  of 
assessment  techniques  that  are  well  known. 
The  NORM  methodology  features  a  set  of 
flexible  standards  that  are  adjusted 
according  to  the  relative  difficulty  of  the 
mission. 

NORM  was  applied  for  field  evaluation  of 
the  Semi-Automatic  Ground  Environment 
(SAGE)  system.  In  dealing  with  any 
man-machine  system,  one  must  ask,  "What  is 
the  system  trying  to  accomplish?"  and 
"What  available  data  will  adequately 
reflect  system  performance?"  Answers  to 
these  questions  will  lead  to  development  of 
suitable  criterion  measures.  The  quality 
of  the  criteria  is  the  single  most 
important  element  in  determining  the 
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meaningfulness  of  the  evaluation.  The 
overall  objective  of  the  SAGE  system  was 
stated  as  "neutralize  as  many  invaders  as 
possible  a:  quickly  as  possible  as  far  out 
as  possible." 

2.2  Once  the  criterion  measures  have  been 
defined,  the  developers  of  the  systemetric 
model  must  try  to  account  for  the  portion 
of  performance  variance  that  is 
attributable  to  mission  difficulty.  To  do 
this,  one  must  determine  the  mission 
characteristics  that  are  most  likely  to 
contribute  to  mission  difficulty.  For  the 
SAGE  system,  the  pertinent  groups  of 
mission  difficulty  variables  were  found  to 
be  (1)  radar  variables,  (2)  invader 
variables,  and  (3)  operational  environment 
variables. 

3.2  The  SAGE  system  overall  objective  was 
translated  into  three  quantifiable 
criterion  measures:  (1)  percentage  fakers 
killed,  (2)  faker  life  time  in  system's  air 
space,  (3)  depth  of  penetration. 

The  above  three  measures  were  supplemented 
by  other  measures  concerning  explicit 
system  functions,  including  (1)  detection 
latency,  (2)  unassociated  time 
non-conformance  of  display  symbology  and 
raw  video,  (3)  interception  time,  and  (4) 
tactical  action  latency. 

M.1  To  develop  more  comprehensive  criteria  of 
effectiveness,  the  performance  measues  are 
factor-analyzed  using  the  principal 
components  methods.  The  first  two  factors 
that  emerge  are  termed  "Weapon  Performance" 
(defined  by  the  measures:  Tactical  Action 
Latency;  Interception  Time;  Depth  of 
Penetration)  and  "Air  Surveillance 
Performance"  (defined  by  five  different 
measures  of  air  surveillance).  These  two 
factors  account  for  about  76%  of  the 
observed  variation  in  performance.  They 
are  more  reliable  than  any  of  the 
individual  measures,  and  they  have 
intrinsic  face  validity. 
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5.1  The  performance  measures  are  collected  from 
simulated  air  defense  missions  performed  at 
SAGE  direction  centers.  The  data  are 
obtained  from  operational  recording  tapes 
containing  a  history  of  all  relevant 
activities  taking  place  during  a  mission. 

5.2  Initial  statistical  procedures  in  NORM 
focus  on  basic  correlations  between  each 
criterion  measure  and  each  mission 
difficulty  variable,  and  on  the  relative 
independence  of  the  candidate  predictor 
variables.  Following  this,  multiple 
regressions  are  run  for  each  measure  using 
selected  sets  of  mission  difficulty 
variables  as  independent  variables. 

Overall,  about  50X  of  the  variance  of 
criterion  performance  is  accounted  by  the 
presently  available  mission  difficulty 
variables. 


5.3  The  criterion  research  in  SAGE  has  resulted 
in  relevant,  quantifiable  measures  of 
system  and  functional  performance.  The 
creation  of  these  measures  has  led  to  a 
meaningful  procedure  for  evaluating 
man-machine  performance  at  the  direction 
center  level. 


Siegel,  A. I.  Human  performance  reliability — Its  measurement  and  impact  on 
system  reliability.  ASHE  Publication,  78-DE-17,  New  York:  The  American 
Society  of  Mechanical  Engineers,  1978. 
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This  paper  discusses  methods  for  measuring 
human  performance  reliability  and  methods 
for  integrating  human  performance  reliabil¬ 
ity  with  equipment  reliability  to  derive  a 
measure  of  total  system  reliability. 

Emphasis  was  placed  on  a  computer  simula¬ 
tion  model.  The  data  presented  show  the 
model  to  possess  validity  and  utility. 

The  model  in  its  current  form  is  basically 
a  sequential  processor  which  incorporates 
human,  equipment,  and  environmental  fac¬ 
tors.  Three  categories  of  independent 
variables  are  considered: 
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(1)  Personnel  (quantity,  category  types, 
goals,  physical  properties,  perform¬ 
ance  attributes). 

(2)  Mission  (composition,  duration, 
environment,  elements/tasks). 

(3)  Equipment  (quantity,  capability, 
performance/status) . 

The  dependent  variables  are  the  Output 
Measurements  (mission  effectiveness,  time 
utilization,  personnel,  and  report 
frequency) . 
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The  purpose  of  this  study  was  to  develop 
and  demonstrate  an  economical  and  practical 
method  for  measuring  technical  proficiency. 

Of  particular  interest  was  the  readiness  of 
Naval  electronically-oriented  technical 
personnel  (reserve  and  regular)  for 
completing  their  assigned  mission. 

The  study  attempted  to  answer  such 
questions  as:  What  is  the  current  level  of 
effectiveness  of  maintenance  personnel  in  a 
given  rating,  ship  or  squadron;  why  is  it 
low  or  high;  what  specific  job  skills  need 
improvement?  Also,  an  attempt  was  made  to 
determine  the  need  for  additional  training 
and  to  compare  the  effectiveness  levels 
between  various  ratings,  ships,  and 
squadrons. 

The  method  relied  largely  on  a  personnel 
reliability  index  modeled  after  an 
equipment  reliability  index.  Specifically, 
the  personnel  reliability  index  was  based 
on  the  compounding  of  probability  of 
successful  performance  values  for  each  of 
eight  factorially  derived  electronic  job 
dimensions.  A  second  instrument  was  also 
administered.  This  instrument  was  based  on 
a  Guttman-scaled  checklist  and  yields  an 
absolute  measure  of  performance. 


6.  Further  Research  Areas 

6.1  Measurement  System 
Limitations 

6.2  Research  Potentials/ 
Priorities 

6.3  Research  Planning 
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5.4  It  was  held  that  the  methods  and  techniques 
employed  a  series  of  psychometric  and 
measurement  criteria.  Statistically 
significant  differences  were  evidenced 
among  the  derived  indices  for  the  Naval 
ratings  and  job  factors  involved  but  not 
among  the  ships  and  squadrons  sampled. 

6.2  The  results  and  experience  of  the  study 
were  interpreted  as  indicating  that  the 
methods  and  data  treatment  techniques 
employed  possess  sufficient  merit  for  their 
purpose  to  warrant  trial  use  on  a  larger 
scale. 
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content  analytic  approach  in  Army  field  system  data  organization  (ARI  Tech. 
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1.2  The  logic,  methods,  and  results  of  a  study 
into  the  derivation  of  a  content  analytic 
approach  to  Army  field  system  data  organiza¬ 
tion  are  described.  The  first  steps  of  the 
program  involved  a  linguistic  analysis  of  a 
set  of  battlefield  messages  and  multidimen¬ 
sional  scaling  analysis  of  a  sample  of 
messages  representing  all  information 
classes  yielded  by  the  linguistic  analysis. 
Four  multidimensional  scaling  analyses  were 
completed.  Each  was  based  on  the  linguis¬ 
tic  similarity  perceptions  of  one  of  four 
experienced  battlefield  analysts.  High 
agreement  was  found  among  the  factorial 
structures  yielded  by  the  data  of  each 
battlefield  analyst.  Accordingly,  an  over¬ 
all  analysis  was  completed  and  15  factors 
were  identified  as  representing  the  percep¬ 
tual  substrate  of  the  Army  field  informa¬ 
tion  linguistic  system.  On  the  basis  of 
the  derived  factors,  a  battlefield  language 
taxonomy  was  developed.  The  taxonomy  was 
tested  in  two  separate  field  oriented  exper¬ 
iments. 


4.  Planning  Components 
of  the  Process 

4.1  Analytic  Methods 

4.2  Parameter  Determinations 

4.3  Apparatus  for  Testing 

4.4  Personnel  for  Testing 

4.5  Test  Plans 

5.  Application  Components 
of  the  Process 

5. 1  Test  Execution 

5.2  Data  Analysis 

5.3  Findings  Interpretation 

5.4  Conclusions  and 
Recommendations 


The  results  of  these  experiments  indicated 
that  intelligence  analysts  can  classify 
messages  reliably  within  the  taxonomy  and 
that  they  can  reliably  use  the  taxonomy  for 
inquiry  purposes.  Moreover,  adequate 
subjective  confidence  was  expressed  by  the 
analysts  in  the  use  of  the  system. 

Finally,  a  computer  system  for  automatic 
classification  of  battlefield  messages  is 
presented . 


6.  Further  Research  Areas 

6.1  Measurement  System 
Limitations 

6.2  Research  Potentials/ 
Priorities 

6.3  Research  Planning 
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This  study  investigated  the  utility  of  the 
human  transfer  function  as  an  indicator  of 
operator  status. 

An  analysis  of  previously  acquired  data 
from  an  experiment  into  the  human  transfer 
function  was  conducted.  In  that 
experiment,  three  subjects  performed  a 
sequence  of  tasks,  including  a  compensatory 
tracking  task,  that  kept  them  occupied 
overnight.  The  results  of  the  study  were 
not  in  accordance  with  the  pre-experimental 
hypothesis,  in  that  the  two  indicators 
derived  were  not  found  to  decrease 
uniformly  as  the  period  of  sleep 
deprivation  increased.  The  present  set  of 
studies  expanded  on  these  previous  data  by 
presenting  a  reanalysis  of  the  previous 
data  and  by  describing  the  rationale, 
method,  and  results  of  a  new  study  into  the 
effects  of  operator  overload  on  the 
transfer  functions. 

The  study  addressed  operator  loading  in 
man-machine  systems. 

Measurements  of  tracking  records  were 
obtained . 

These  measurements  were  made  to  the  nearest 
quarter  millimeter. 

Data  for  three  minutes  of  each  selected 
hourly  sample  were  transcribed  at  0.10 
second  intervals.  This  yielded  1800  data 
points  for  each  input  and  1800  for  each 
output  position  signal  sample.  The  first, 
second,  fourth  and  sixth  hours  of  tracking 
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performance  were  selected  for  analysis. 
Frequency  analysis  was  employed  in  data 
analysis. 

4.3  An  oscilloscope  display  and  a 
tracking-control  were  used  in  this 
experiment.  Programming  and  recording 
equipment  were  also  utilized. 

4.4  Four  male  college  students  were  subjects  in 
this  study.  They  were  paid  for  their 
participation. 

4.5  The  experimental  subjects  tracked 
continuously  for  eight  hours  between  8  am 
and  4  pm.  No  breaks  were  allowed.  Samples 
of  their  performance  were  recorded  for  the 
last  five  minutes  of  each  hour.  The 
subjects  were  unaware  how  much,  if  any  of 
their  performance  was  being  recorded. 
Transcription  of  the  data  involved 
measuring  the  displacement  of  the  input  and 
output  signals  as  recorded  on  the  ink 
writing  oscillograph. 

5.3  The  results  of  this  3tudy  were  interpreted 
to  suggest  that  the  spectral  analytic 
technique  possesses  considerable  potential 
as  an  objective,  behavioral,  diagnostic 
method  for  on-line  assessment  of  operator 
status  in  man-machine  systems  involving 
perceptual  motor  behavior. 
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This  report  describes  a  matrix  method  for 
the  evaluation  of  training. 

Training  programs  in  general  were  the 
object  of  the  evaluation. 

The  mission  of  the  training  program  was 
described  as  the  preparation  of  students 
for  the  Jobs  performed  after  training. 

"Suitability"  for  the  Job  was  used  as  a 
basis  for  training  evaluation.  "Suitabil¬ 
ity"  was  defined  as  the  training  graduates' 
ability  to  do  the  tasks  involved  in  the 
job. 
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Matrix  solutions  were  described  which  yield 
three  indices,  each  reflecting  a  different 
aspect  of  the  comparison  between  the  skills 
of  the  trained  man  and  the  job's  require¬ 
ments. 

The  training  program  is  deemed  effective  if 
the  graduate  carries  out  the  duties  of  his 
Job  proficiently. 

Specific  measures  of  job  performance  can 
indicate  the  effectiveness  of  a  training 
program. 


5.  Application  Components 
of  the  Process 

5.1  Test  Execution 

5.2  Data  Analysis 

5.3  Findings  Interpretation 

5.4  Conclusions  and 
Recommendations 


6.  Further  Research  Areas 

6.1  Measurement  System 
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PH  orlties 

6.3  Research  Planning 
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3.5  In  this  study  a  specific  evaluative  scheme 
was  developed  for  summarizing  suitability 
and  the  application  of  the  scheme  to  data 
previously  collected  on  technicians  in  four 
Naval  ratings.  The  method  for  accom¬ 
plishing  these  tasks  is  the  Technical 
Behavior  Checklist  (TBCL).  These  check¬ 
lists  are  comprehensive  detailed  compila¬ 
tions  of  the  tasks  performed. 

4.1  Various  responses  to  the  TBCL  by  the 
supervisor  indicating  the  level  of 
proficiency  of  the  ratee  were  assigned 
different  score  values.  The  average 
proficiency  level  of  each  task  was  computed 
-  -  determined  by  the  mean  of  these  scores. 
An  index  of  importance  of  each  task  was 
developed  from  the  TBCL  and  the  supervisor. 
These  importance  ratings  were  weighted  and 
the  task's  importance  for  classification  in 
the  matrix  was  a  combination  of  its 
frequency  of  performance  and  its 
criticality  to  the  mission.  From  this 
information  three  indices  were  developed. 

(1)  The  training  index;  (2)  an  "over¬ 
training"  index;  (3)  an  "undertraining" 
index.  The  training  index  reflects  how 
closely  the  average  graduate  fits  the  model 
of  high  proficiency  on  relatively  important 
tasks  and  low  proficiency  on  relatively 
unimportant  tasks.  The  overtraining  index 
indicated  the  extent  to  which  graduates 
were  highly  proficient  even  on  unimportant 
tasks.  Important  tasks  that  technicians 
performed  with  relatively  low  proficiency 
constituted  the  undertraining  index. 

4.4  The  subjects  of  this  study  were: 

(1)  Aviation  Machinist  Mates  (Turbo  Jet) 

(2)  Air  Controlman  (Tower) 

(3)  Parachute  Rigger  (Survivalman) 

(4)  Aviation  Electronics  Technician  (Radar) 
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4.5  In  a  previous  study  a  Technical  Behavior 
Checklist  was  developed  for  four  naval 
ratings.  This  checklist  was  a  detailed 
comprehensive  checklist  of  the  tasks 
performed  in  that  rating.  For  this  study  a 
supervisor  was  asked  to  indicate  the 
proficiency  level  of  the  man  he  was  rating 
in  terms  of  how  much  supervision  and  the 
number  of  checkouts  required. 

5.3  The  technicians  in  the  four  ratings  studied 
were  found  generally  to  be  suitably 
trained. 

5.4  The  matrix  method  is  feasible  and  simple  to 
use  provided  task  proficiency  measures  and 
an  index  of  task  importance  are  available. 
The  use  of  this  method  provides  meaningful 
results.  Its  use  should  provide  an 
increased  understanding  of  the  effects  of 
training  and  is  one  basis  for  changing 
training  emphasis. 
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silliness?  In  G.  W.  Levy  (Ed.),  Symposium  on  applied  model  of  man-machine 
systems  performance  (NR69H-59D.  Columbus,  OH:  North  American  Aviation, 
November  1968.  (AD-697  939). 
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The  paper  discusses  advantages  and 
disadvantages  of  stochastically  oriented 
digital  computer  simulation  models  and 
their  applications  to  predicting  the 
effects  of  the  human  on  the  output  of 
complex  systems.  The  advantages  of 
simulation  modeling  are  given  as  follows: 


(1) 


It  necessitates  a  rigorous, 
insight-providing  analysis  of  the 
human  and  machine  elements  and  their 
interactions. 


(2) 


It  provides  a  means  of  representing 
the  complex  human  organism  to  a 
degreee  of  realism  not  available 
through  deterministic  mathematical 
formulations. 


(3) 


It  can  cost  less  than  either  physical 
simulation  or  laboratory 
experimentation. 


(U) 


It  allows  the  testing  of  hypothetical 
systems  and  of  hypothetical  operating 
procedures. 


(5) 


It  can  show  the  effects  of 
interactions  on  system  function,  e.g., 
the  interaction  between  operator  skill 
and  workday  length. 


(6)  It  can  be  useful  in  establishing 

training  requirements  and  areas  for 
training  emphasis. 
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(7)  It  enables  us  to  deal  with  a  complex¬ 
ity  that  cannot  be  managed  by  other 
techniques. 

It  is  also  pointed  out  that  there  are 
adverse  aspects  to  digital  simulation  which 
could  serve  to  limit  its  potential.  These 
include: 

(1)  There  is  a  tendency  to  confuse  the 
model  on  which  the  simulation  is  based 
with  a  theory,  and  hence  to  employ 
inappropriate  standards  and  criteria 
to  Judge  the  model  and  the  simulation. 

(2)  Digital  modeling  depends  on  the  data 
and  methods  of  a  number  of 
disciplines,  and  this  can  lead  to 
complexity  and  confusion. 

(3)  The  necessary  data  may  be  unavailable 
or  of  poor  quality  (garbage  in, 
garbage  out) . 

(4)  There  are  equally  damaging  but 
opposite  points  of  view  that  impede 
serious  research  by  computer 
simulation;  one  is  that  attempts  to 
simulate  man  via  a  computer  are 
antireligious  or  at  least  degrading; 
the  other  is  that  a  computer's  results 
are  automatically  sacred  and  of 
unquestionable  precision. 

(5)  The  argument  exists  that  behavioral 
science  does  not  possess  powerful 
enough  theory  for  model  building. 

5.4  It  is  concluded  that  digital  simulation  is 
an  emergency  tool  for  the  behavioral 
sciences. 
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1.1  The  integration  of  Human  Reliability  (HR) 
and  Equipment  Reliability  (ER)  data  into  a 
single  comprehensive  model  for  predicting 
System  Reliability  (SR)  has  been  one  of  the 
major  concerns  of  Navy  system  planners. 

1.3  The  present  study  attempted  to: 

(1)  Extend  and  strengthen  a  previously 
developed  model  for  simulating  the 
acts  and  behaviors  of  the  operators  of 
an  intermediate  size  system  to  include 
a  greater  number  of  options  to  the 
model  user. 

(2)  Evolve  the  model  into  one  which 
produces  reliability  oriented  metrics 
for  both  humans  and  equipment  on  both 
an  event  and  overall  system  level. 

(3)  Conduct  an  initial  series  of 
sensitivity  tests  relating  the  new 
variables  and  parameters  to  those 
already  present  in  the  model. 


A.  Planning  Components  3.I 

of  the  Process 

4.1  Analytic  Methods 

4.2  Parameter  Determinations 

4.3  Apparatus  for  Testing 

4.4  Personnel  for  Testing 

4.5  Test  Plans 

5.  Application  Components 
of  the  Process 
5.1  Test  Execution 
5-2  Data  Analysis 

5.3  Findings  Interpretation 

5.4  Conclusions  and 
Recommendations 


The  variables  selected  for  measurement  in 
the  present  document  included: 


(1) 

Physical  capability 

-  working  pace 

(2) 

Competence 

-  level  of 
aspiration 

(3) 

Fatigue 

-  psychological 
stress 

6.  Further  Research  Areas 

6.1  Measurement  System 
Limitations 

6.2  Research  Potentials/ 
Priorities 

6.3  Research  Planning 


(4)  Physical 

incompatibility 


-  confidence 


4.1  Data  were  analyzed,  in  part,  via  the  mean 
and  standard  deviation. 
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two-stage  approach  to  experimentation: 
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variables  and  second,  to  relate  them 
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The  report  provides  an  overall  skeletal 
structure  of  the  Operational  Test  and 
Evaluation  (OTAE)  process.  From  the 
structure,  a  step-by-step  procedure  is 
derived.  Steps  can  be  summarized  as 
follows: 


(1)  Review  of  documentation. 

(2)  Formulation  of  test  objectives. 

(3)  Selection  of  applicable  test  concept 

(4)  Measures  of  effectiveness  (MOE). 

(5)  Test  design. 

(6)  Simulations. 

(7)  Data. 

(8)  Range  instrumentation. 

(9)  Test  plan. 

(10)  Conduct  of  test. 

(11)  Data  analysis. 

(12)  Conclusions  and  recommendations. 

(13)  Test  report. 
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2.1  Review  of  documentation  relevant  to  a  given 
system  to  be  tested  is  important  because 
the  test  managers  need  to  have  a  clear 
understanding  of  the  critical  issues,  data 
requirements,  and  test  objectives,  and  reed 
to  be  familiar  with  the  system,  how  it 
operates  and  its  history  in  previous  tests. 

2. 4  Selection  of  a  test  concept  is  a  trade-off 
between  the  magnitude  and  depth  of  testing 
and  the  available  resources.  Factors 
affecting  the  trade-off  may  include: 

(1)  Time. 

(2)  Availability  of  the  (test)  range. 

(3)  Availability  of  personnel. 

(4)  Availability  of  equipment. 

(5)  Risk  of  overrunning  time  due  to  test 

complexity. 

(6)  Data  quality  and  quantity. 

(7)  Cost. 

3.1  OT&E  objectives  are  formulated  to  answer 
critical  questions  and  to  provide  a  basis 
for  making  decisions  affecting  development, 
production,  support,  employment  of  a  system. 
Typical  OT&E  objectives  are: 

(1)  To  observe  the  degradation. 

(2)  To  collect  data  on... 

(3)  To  evaluate  ability  to  meet  the 
requirement. .. 

3.2  Measure  of  effectiveness  of  an  item  is  a 
parameter  which  evalates  the  extent  of  the 
adequacy  of  the  item  to  accomplish  an 
intended  mission  under  specific  conditions. 
An  MOE  is  a  function  of  Availability  (A), 
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Dependability  (D),  and  Capability  (C). 

MOE's  are  expressed  as  probabilities  since 

A,  D,  C  are  probabilities.  Good  MOE's: 

(1)  Should  be  sensitive  to  all  variables 
affecting  the  item. 

(2)  Should  be  precisely  defined. 

(3)  Should  not  be  overly  broad. 

(4)  Should  be  mutually  exclusive. 

(5)  Should  have  exhaustive  inputs. 

(6)  Should  be  relevant  to  the  mission. 

(7)  Should  have  inputs  relevant  to  the 
design  issues. 

(8)  Should  be  expressed  in  terms 
meaningful  to  the  decision  maker. 

(9)  Should  have  inputs  that  are  measurable. 

(10)  And  its  inputs  should  be  quantifiable 
it  at  all  possible. 
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The  cost  and  complexity  of  modern  military 
systems  have  given  rise  to  the  development 
of  a  technology  that  is  referred  to  as 
system  and  cost  effectiveness  analysis. 
Until  recently,  however,  only  limited 
effort  has  been  directed  toward  the 
specification  of  the  contribution  made  by 
system  personnel  as  well  as  human  error  in 
a  system's  overall  effectiveness. 

TEPPS  (Technique  for  Establishing  Personnel 
Performance  Standards)  was  designed  to  meet 
two  primary  objectives: 

(1)  To  derive  specific  personnel 
performance  standards  with  definable 
relations  to  system  effectiveness 
requirements. 

(2)  To  determine  the  influence  on  system 
effectiveness  of  performance  levels 
that  deviate  from  established 
performance  standards. 

A  system  was  a  set  of  personnel-equipment 
functional  units  whose  collective  purpose 
is  to  achieve  a  particular  goal.  Differen¬ 
tiating  between  systems  and  subsystems  is 
arbitrarily  based  since  most  systems  can  be 
defined  as  subsystems  when  referenced  to 
larger  overall  systems  of  which  they  are  a 
part.  What  is  important  is  the  relation¬ 
ship  of  a  given  system’s  goals  with  respect 
to  those  of  another.  Thus,  the  effective¬ 
ness  of  a  given  system  should  be  evaluated 
with  respect  to  the  parent  system.  A 
system  can  be  assumed  to  have  been  100% 
effective  if  it  performed  up  to  its  maximum 
capability,  regardless  of 
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whether  or  not  it  was  subsequently  de¬ 
stroyed.  Thus  capability  is  an  important 
factor  contributing  to  establishing  system 
effectiveness  requirements  and  the  evalu¬ 
ation  of  system  effectiveness. 
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Through  the  years,  technological  advances 
have  provided  military  systems  with 
increased  capability.  Planners  have 
subsequently  devised  techniques  or  tools  to 
resolve  problems  more  quickly;  however, 
many  such  tools  are  individualistic  in  that 
they  have  been  independently  developed  and 
employed  by  individuals  through  their  own 
unique  experiences.  For  that  reason  TEPPS 
(Technique  for  Establishing  Personnel 
Performance  Standards)  was  developed. 
Overall,  TEPPS  is  a  3et  of  procedures  for 
gathering  information  about  a  system  and 
for  developing  a  qualitative  and/or 
quantitative  model  of  that  system.  The 
utility  and  accuracy  of  TEPPS  depends  on 
such  information  as  system  descriptive 
data,  operational  requirements  data,  and 
human  capability  data. 

A  graphic  representation  of  the  major  steps 
in  applying  TEPPS  is  presented  in  Figure  1. 

The  system  employed  was  an  Anti-Air  Warfare 
Combat  Information  Center  (CIC)  typically 
employed  on  a  picket  ship  involved  in  fleet 
defense.  The  CIC  system  is  composed  of 
three  functions; 

(1)  The  gathering  and  display  of  target 
information. 

(2)  Actions  of  the  command  officer 
(evaluator)  on  that  information. 

(3)  The  Air  Intercept  Controller's  (AIC) 
vectoring  of  a  Combat  Air  Patrol  (CAP) 
fighter. 
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Figure  1.  Representation  of  Major  Steps 
in  Applying  TEPPS 
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The  purpose  of  this  report  is  to  present 
a  systematic  treatment  of  the  major  con¬ 
siderations  in  the  measurement  of  advanced 
flight  vehicle  crew  proficiency  in  syn¬ 
thetic  ground  environments. 

Measurement  in  the  training  context  serves 
two  main  functions.  One  is  prediction  of 
how  well  an  individual  or  team  can  be  ex¬ 
pected  to  perform  under  some  specified  con¬ 
ditions  anticipated  in  the  future.  The 
second  is  measurement  of  present  knowledge 
or  level  of  proficiency  in  some  area  or 
task. 

It  is  always  a  difficult  problem  to  define 
mission  purpose.  Recognizing  that  a  pre¬ 
cise  definition  will  seldom  or  ever  be 
spelled  out,  some  statement  of  mission 
purpose  must  nevertheless  be  available. 

Such  a  statement  represents  a  starting  point 
insofar  as  determining  the  criteria  for 
measurement  and  for  suggesting  what  specific 
tasks  are  most  critical. 
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4.5  Test  Plans 

5.  Application  Components 
of  the  Process 

5.1  Test  Execution 

5.2  Data  Analysis 

5.3  Findings  Interpretation 

5.4  Conclusions  and 
Rec  onxnend  at  i  on  s 

6.  Further  Research  Areas 

6.1  Measurement  System 
Limitations 

6.2  Research  Potentials/ 
Priorities 

6.3  Research  Planning 


Seldom  can  one  obtain  direct  measures  of  the 
ultimate  criteria  for  a  system.  It  is 
usually  necessary  to  select  some  actual 
criterion  which  in  practice  is  an  approxi¬ 
mation  of  the  ultimate  one.  In  general,  a 
good  criterion  is  one  that  is  both  reliable 
and  relevant.  Reliability  implies  that 
proficient  or  successful  performance  will 
not  vary  widely  because  of  chance  factors. 
Relevancy  refers  to  the  validity  of  the 
actual  criteria.  The  actual  criterion  is 
relevant  to  the  extent  that  it  approximates 
the  ultimate  criterion. 
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The  multidimensional  nature  of  proficient 
performance  will  often  require  that  sev¬ 
eral  jriteria,  all  of  which  are  relevant 
for  a  particular  activity,  be  used.  In 
such  cases,  it  may  be  desirable  to  combine 
these  several  criteria  into  a  single  com¬ 
prehensive  one.  Combining  subcriteria  into 
a  single  composite  will  usually  involve 
assigning  relative  weights  to  the  individual 
criterion.  These  weights  must  often  be 
determined  on  the  basis  of  expert  opinion 
since  it  is  seldom  possible  to  determine 
statistically  the  intercorrelation  or  over¬ 
lap  between  subcriteria.  A  measured  rela¬ 
tionship  between  the  actual  and  ultimate 
criteria  is  even  more  rare.  Subcriteria 
should  be  weighted  according  to  their  rele¬ 
vance,  and  those  which  overlap  factors  in 
another  subcriteria  should  receive  a  low 
weight.  Subcriteria  which  are  more  reliable 
should  be  weighted  more  than  those  suMect 
to  error. 

3-2  Both  overall  and  diagnostic  measures  of 

performance  are  required  in  any  systematic 
effort.  Overall  measures  refer  to  global 
indices  of  behavior  associated  with  large 
aspects  of  performance  such  as  occur  in 
mission  segments  and  complete  missions. 
Diagnostic  measures  are  quite  specific, 
identifying  certain  aspects  or  elements  of 
a  job  or  performance  in  specific  skill  areas. 

In  any  extensive  evaluation  effort,  subjec¬ 
tive  and  objective  measures  might  well  be 
used.  Generally,  objective  measures  permit 
measurement  relatively  independent  of  the 
observer  and  with  a  high  degree  of  relia- 
ability.  However,  insistence  on  complete 
objectivity  tends  to  result  in  omission  of 
a  variety  of  critical  job  components  because 
of  the  inability  to  measure  them  objectively. 
Subjective  techniques  also  have  limitations. 
The  evaluation,  generally,  is  dependent  upon 
the  characteristics  of  the  observer  and 
agreement  between  independent  observers  is 
not  high.  The  use  of  such  observers  can 
introduce  a  set  of  biases  into  the 
observations . 
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Much  of  what  is  stated  about  combining 
criteria  also  applies  to  combining  perfor¬ 
mance  measures.  However,  in  the  latter 
case  it  is  usually  possible  to  determine 
the  interrelationship  between  each  measure 
and  the  criterion.  Such  information  is 
important  in  developing  a  single  combined 
proficiency  score. 

A  single  overall  proficiency  score  has  the 
desirable  characteristic  of  providing  one 
overall  proficiency  index.  However,  a 
point  to  consider  in  determining  whether 
or  not  to  combine  several  performance 
measures  is  that  a  single  index  will  no 
longer  reflect  performance  of  the  individ¬ 
ual  measures.  Where  at  least  a  minimum 
amount  of  proficiency  with  respect  to  a 
particular  measure  is  critical  to  an  over¬ 
all  activity,  it  may  be  more  appropriate 
to  treat  the  several  measures  in  terms  of 
a  profile.  This  will  preserve  the  indi¬ 
vidual  measures  while  allowing  for  a 
simultaneous  viewing  of  the  measured  results 

It  is  most  important  that  any  system  for 
structuring  measurement  of  proficiency  re¬ 
quires  as  an  integral  part  a  systematic 
classification  of  behavior  which  can  be 
present  within  the  system.  Underlying  the 
development  of  this  report  is  the  assumption 
that  behavior  can  be  analyzed  in  terms  of 
basic  components,  that  these  components  can 
be  conceptually  identified  in  a  way  that  is 
convenient  and  agreeable  to  people  and  that 
there  are  specific  measurement  techniques 
which  appropriately  go  with  various  behav¬ 
ioral  components.  The  behavior  classifica¬ 
tion  is  operationally  defined  and  structured 
in  a  way  meaningful  for  measurement.  Four 
levels  of  job  analysis  are  presented  under 
which  can  be  subsumed  the  varieties  of  opera 
tional  events  and  behaviors  predicted  for 
the  system.  The  scheme  is  able  to  accommo¬ 
date  both  diagnostic  measures  relating  to 
elemental  tasks  as  well  as  to  more  global 
measurements  relating  to  overall  system 
performance.  The  classes  are  as  follows: 
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Elemental  tasks 


Level  II  Complex  tasks  irvolving 

composite  learning  activities 

Level  III  Mission  segments 

Level  IV  Combined  mission  segments 

The  classification  begins  with  the  simplest 
level  of  analysis  and  proceeds  to  the  most 
complex  and  is  so  structured  that  each 
succeeding  level  of  complexity  is  made  up 
of  recognizable  behavioral  units  of  the 
proceeding  level. 

A  sequence  of  logical  steps  is  presented 
concerned  with  setting  up,  obtaining  and 
reporting  proficiency  measures  of  individuals 
and  crews.  With  these  procedures,  the  user 
is  provided  guidance  for  the  design  of  a 
measurement  system. 

Step  1  -  Conduct  a  system  and  job  analysis 

Step  2  -  Identify  important  and  critical 
tasks 

Step  3  -  Determine  performance  requirements 
for  important  tasks 

Step  -  Select  and  obtain  measurements 

appropriate  for  the  behavior  to  be 
evaluated 

Step  5  -  Make  decisions  on  recording  the 

measurement  data  and  combining  the 
results 


*  •  * 
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PIMO  is  an  acronym  for  a  project  entitled 
Presentation  of  Information  for  Maintenance 
and  Operation.  The  purpose  of  the  project 
is  to  develop  a  new,  improved  approach  for 
presenting  the  technical  data  used  by  Air 
Force  maintenance  technicians.  For  test 
purposes,  the  project  focused  on 
maintenance  operations  for  the  C-141A  jet 
cargo  aircraft. 

The  staff  realized  that  improvements  in 
maintenance  information  presentation  had  to 
be  expressed  in  terms  of  the  object  system, 
i.e.,  the  C-141A.  The  value  of  changes  in 
performance  at  the  support  level  must  be 
evaluated  in  terms  of  object  system 
performance  and/or  cost. 

Some  means  had  to  be  devised  which  would 
relate  changes  in  maintenance  performance 
to  changes  in  C-141A  effectiveness.  The 
means  employed  was  the  Aircraft  Maintenance 
and  Effectiveness  Simulation  model  (AMES). 

The  AMES  model  was  constructed  to  include 
measures  of  functional  reliability  and 
alternative  personnel  utilization,  and  was 
used  to  establish  payoffs  in  terms  of 
increased  aircraft  utilization  and  cost 
savings  which  could  be  compared  to  the  cost 
of  maintenance  information  system 
improvements. 


6.  Further  Research  Areas 

6.1  Measurement  System 
Limitation* 

6.2  Research  Potentials/ 
Priorities 

6.3  Research  Planning 
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The  goal  of  this  study  was  to  provide 
objective,  quantitative  methods  of 
measuring  pilot  workload. 

The  system  being  measured  was  a  pilot  and  a 
simulated  tracking  task. 

The  experiment  took  place  in  ?  laboratory 
situation  and  physical,  psychological,  and 
environmental  conditions  were  kept  as 
constant  as  possible. 

The  attributes  measured  were  physiological 
and  performance  dimensions  over  a  range  of 
visual  motor  tracking  tasks,  and  the 
subject's  reserve  capacity. 

Electrophysiological  measurements  were  made 
and  a  sensitive,  nonloading  measure  of 
reserve  capacity  was  determined. 

A  Measured  Workload  Index  (MWI)  and  a 
Physiological  Workload  Index  (PWI)  were 
extracted.  A  workload  index  based  on  the 
pilot's  physiological  response  to  a 
simulated  tracking  task  was  evolved. 
Important  steps  in  this  approach  included: 

(1)  Validation  of  a  sensitive,  nonloading 
secondary  task. 


5.  Application  Components  (2) 

of  the  Process 

5.1  Test  Execution 

5.2  Data  Analysis 

5.3  Findings  Interpretation 

5. 4  Conclusions  and  (3) 
Recommendations 

6.  F'urther  Research  Areas 

6.1  Measurement  System 
Limitations 

6.2  Research  Potentials/ 

Priorities 

6.3  Research  Planning 


Collection  of  physiological  and  per¬ 
formance  data  over  a  range  (easy  to 
land)  visual  motor  tracking  tasks. 

Extraction  of  any  potentially  meaning 
ful  features  from  the  analog 
physiological  data. 
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(6)  Simultaneous  computation  of  the 
workload  index  and  the  best  linear 
predictor  from  the  subset  of  features. 

(7)  Validation  of  this  predictor. 

The  entire  study  was  structured  to  provide: 

(1)  A  sensitive,  non-loading  measure  of 
reserve  capacity. 

(2)  An  unencumbering  reliable  measure  of 
the  psychological  state.  An  important 
measure  of  the  success  of  this  study 
was  the  degree  to  which  the  MWI  and 
the  PWI  agreed  across  the  randomly 
presented  243  -  four-minute  trials. 

Three  direct  measures  of  reserve  capacity 

were  provided: 

(1)  Miss  Rate  —  %  of  error  in  responding 
to  secondary  task. 

(2)  Response  time  -  -  average  time  from 
secondary  task  stimulus  onset  to 
response. 

(3)  Subjective  Rating  -  -  pilots’ 
evaluation  of  task  difficulty.  AB137. 

The  salient  features  of  this  study  include: 

(1)  A  simple,  sensitive,  nonloading 
secondary  task. 

(2)  A  subjective  rating  which  agrees  with 
other  secondary  task  measures  but  with 
less  intersubject  variance. 


(3)  A  multichannel  physiological 

monitoring  and  recording  system. 
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(4)  Automatic  feature  extraction  software 
which  transforms  the  analog  data  base 
into  meaningful  features. 

(5)  Very  good  separation  results  using  a 
pattern  recognition  system,  assuming 
the  data  to  represent  a  two-class 
problem. 

(6)  Use  of  simultaneous  least-squares 
prediction  to  arrive  at  statistically 
significant,  validated,  workload  index 
and  the  physiological  features  which 
best  predict  it. 
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W.L.  Definition  of  requirements  for  a  performance  measurement  system  for  C-5 
aircrew  members  (AFHRL-TR-TS-S1*) .  San  Diego,  CA:  Log  icon  Inc.,  October  1978. 
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1.1  General  System 
Measurements 

1.2  System  Taxonomy 
Model  (ST>1) 

1.3  Overall  Conceptual 
Process  Model  (CPM) 


1.3  This  study  identifies  and  defines  C-5 
aircrew  tasks  and  performance. 

2.1  The  system  addressed  was  the  C-5  aircraft 
and  aircrew. 

2.2  The  mission  definition  was  the  effective 


Contextual  Components 
of  the  Process 

2.1  System  Definition 

2.2  Mission  Definition 

2.3  Environment  Definition 

2.4  General  Constraints 

2.5  Performance 
Requirements,  Ultimate 

2.6  Performance 
Criteria,  Ultimate 

Analytic  Components 


operation  of  the  aircraft  on  a  typical 
mission 

Aircrew  p  rformance  was  selected  for 
measurement. 

A  candidate  set  of  performance  measures  was 
developed  to  assess  aircrew  proficiency. 
This  was  accomplished 
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(3) 
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Determination  and  segmentation  of  a 
representative  mission  profile. 

The  determination  of  aircrew 
activities  within  each  segment. 

The  determination  of 
mission-essential/critical  aircrew 


Planning  Components 
of  the  Process 

4.1  Analytic  Methods 

4.2  Parameter  Determinations 

4.3  Apparatus  for  Testing  3*5 

4.4  Personnel  for  Testing 

4.5  Test  Plans 

Application  Components 
of  the  Process 

5.1  Test  Execution 

5.2  Data  Analysis 

5.3  Findings  Interpretation 

5.4  Conclusions  and 
Recommendations 


tasks  and  duties  amongst  these  aircrew 
activities. 

The  C-5  inflight  performance  measurement 
can  be  obtained  through  several  methods. 

It  appears  that  the  C-5  MADAR  system  which 
routinely  monitors  approximately  19  parame¬ 
ters  and  has  the  ability  to  interrogate  a 
number  of  key  switch  positions  will  provide 
a  feasible  basis  for  interfacing  the 
inflight  measurement  system  with  the  C-5 
components. 


Further  Research  Areas 

6.1  Measurement  System 
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4.3  A  C-5  simulater  was  used  in  this  study. 

4.5  In  Phase  I  candidate  performance  measures 
were  developed.  The  source  of  this 
information  included  documentation, 
interviews,  and  dialogue  with  operationally 
qualified  aircrews.  A  special  purpose 
evaluation  sortie  for  the  C-3  simulater  was 
developed. 

In  Phase  II  efforts  were  developed  toward 
the  definition  of  several  alternative 
configurations  to  meet  the  performance 
measure  requirements  and  provide  MAC  with 
capabilities  of  conducting  the  evaluation. 

A  review  of  existing  system,  documentation, 
and  interviewing  techniques  accomplished 
this  task. 

In  Phase  III,  the  functional  and 
engineering  requirements  for  the  C-5 
performance  measurement  system  were 
described. 
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1.3  This  study  was  undertaken  to  compare  two 
ball  port  designs. 

2.1  The  current  firing  port  weapon  (XM23D/ball 
port  design  for  the  Infantry  Fighting 
Vehicle  (IFV)  was  compared  to  a  new  design. 

2.2  Data  were  collected  on  the  removal  and 
installation  of  the  firing  port  weapons. 

2.6  Critical  to  acceptance  of  the  proposed 

design  was  its  effect  on  the  time  required 
to  dismount  as  part  of  rapid  egress  from 
the  vehicle.  In  addition,  the  weapon 
retention  characteristics  were  considered. 

3.2  Time  required  for  removal  and  installation 
of  the  firing  port  weapon  was  measured. 

4.4  Five  subjects  and  one  test  coordinator 
participated  in  the  experiment. 

4.5  Each  subject  was  trained  to  install  and 
remove  the  weapon  on  each  configuration. 

The  seat  chosen  for  the  experiment  was  at 
the  worst  possible  angle  for  the  tasks. 

Each  subject  performed  six  trials  in 
removing  and  installing  the  weapon  in  each 
design  configuration.  Time  measurements 
were  obtained  by  means  of  a  stopwatch. 
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5.4  Conclusions  and 
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5.4  Average  time  for  removal  of  the  firing  port 
weapon  was  1.4  seconds  for  the  current 
design-used  snap  ring,  3.5  seconds  for  the 
current  design-new  snap  ring,  and  3.^ 
seconds  for  the  proposed  design..  While  the 
current  design-used  snap  ring  is  approxi¬ 
mately  2  seconds  faster  for  weapon  removal 
than  either  of  the  other  two  configu¬ 
rations,  its  condition  (state  of  fatigue  of 
the  snap-ring)  was  probably  unsafe. 


Average  times  for  installation  of  the 
firing  port  weapon  were  8.4  seconds,  6,2 
seconds,  and  3*8  seconds  with  the  current 
design-used  snap  ring,  current  design-new 
snap  ring  and  the  proposed  design, 
respectively.  While  installation  time  is 
less  critical  than  removal  time  it  is  clear 
that  the  proposed  design  provides  an 
advantage  for  ingress  and  rapid  firing  of 
the  weapons  mounted  in  the  ramp. 
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This  proposal  describes  a  study  to 
investigate  and  develop  a  methodology  for 
human  factors  evaluation  of  equipment  using 
field  performance  measures  at  the 
small-group  level.  The  theoretical 
framework  around  which  this  proposed  study 
is  oriented  involves  the  determination  of 
relationships  among  subtasks  and  between 
subtask  outputs  and  the  terminal  task 
output.  It  is  conceptualized  that 
measurement  criteria  should  be  developed 
for  those  subtasks  and  outputs  which  are 
most  highly  correlated  with  the  terminal 
task  output.  Additional  inputs  to  the 
measurement  should  involve  consideration  of 
the  stress  imposed  on  task  performance  by 
various  operational  conditions.  Equipment 
operability  would  be  defined  in  terms  of 
resistance  to  the  performance  degadation 
caused  by  these  stress  conditions. 

The  proposed  process  for  the  development  of 
valid  field  performance  measures  is  as 
follows: 

(1)  Select  tasks  which  manifest  a  range  of 
behaviors  from  complex  to  simple; 
which  are  related  in  the  performance 
of  some  total  system  function;  which 
are  exposed  normally  to  a  variety  of 
operational  conditions. 

(2)  Analyze  tasks  to  describe  the  task 
hierarchy  and  show  interrelationships 
among  tasks;  to  describe  the 
behavioral  functions  which  implement 
performance;  to  determine  the  points 
at  which  load  conditions  arise  for 
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personnel;  to  indicate  contingent 
events  which  may  affect  task  perform¬ 
ance;  to  indicate  the  probable  effect 
on  task  performance  of  introducing 
chemical  agent  protective  equipment; 
to  describe  the  nature  of  stimulus 
inputs,  terminal  outputs  and  potential 
methods  of  task  measurement. 

(3)  Develop  and  administer  a  task 

performance  characteristics  scaling 
test  which  would  depend  on  the 
experience  of  skilled  military 
personnel  to  discriminate  between  the 
correlated  factors  involved  in 
selected  task  performance. 

(iJ)  Develop  predictive  criteria  from  the 
test  data  to  be  validated  in  field 
exercises. 

(5)  Conduct  the  field  exercises;  record 
and  analyze  the  field  data  to 
determine  the  correlation  between  test 
and  field  measurements. 


Tien,  J.M.  Toward  a  systematic  approao  o  program  evaluation  design.  IEEE 
Transactions  on  Systems,  Man,  and  Cybernetics,  September  1979.  SMC— 9 ( 9 ) . 
494-515. 
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In  this  paper,  the  broad  field  of  program 
evaluation,  specifically  evaluation  of 
programs  in  the  public  sector,  is  reviewed. 
Attempts  are  made  to  synthesize  and 
systematize  the  steps  necessary  to  develop 
valid  and  comprehensive  evaluation  designs. 
First,  a  design  framework  is  identified 
which  links  program  characteristics  to 
design  elements  through  an  expanded  set  of 
threats  to  validity.  Second,  the  various 
design  elements  are  grouped  into  five 
systematically  convenient  components, 
including  test  hypothesis,  selection 
scheme,  measures  framework,  measurement 
methods,  and  analytic  techniques.  Third, 
it  is  proposed  that  the  different  types  of 
evaluation  can  be  contained  in  an 
evaluation  taxonomy  composed  of  eight 
measure s-related  classifications. 

It  is  noted  that  there  were  many  ways  of 
classifying  a  program  evaluation  effort: 
by  subject  matter  of  the  evaluation;  by  the 
purpose  of  the  evaluation;  by  the 
methodology  employed  in  the  evaluation;  or 
by  some  other  criterion.  A  number  of  the 
most  commonly  used  evaluation  categoriza¬ 
tions  is  listed  in  terms  of  the  subject 
matter  or  purpose  of  the  evaluation. 

The  authors  state  that  this  paper  should  be 
regarded  as  only  an  initial  step  towards  a 
systematic  approach  to  program  evaluation 
design  while  a  basis  for  further  discussion 
is  provided. 
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1.1  There  is  an  effort  underway  to  develop  the 
framework  of  a  procedure  for  the  collection 
and  analyses  of  data  on  uncertainty  and 
progress  regarding  technical  performance  in 
weapon  systems  development.  The  data 
collected  are  concerned  with  uncertainty 
about  the  characteristics  of  the  component 


2.  Contextual  Components 
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2.1  System  Definition 

2.2  Mission  Definition 

2.3  Environment  Definition 

2.4  General  Constraints 

2.5  Performance 
Requirements,  Ultimate 

2.6  Performance 
Criteria,  Ultimate 


parts  of  the  total  system.  These  data  are 
combined  using  systems  design  relationships 
to  determine  the  uncertainty  about  the 
performance  of  the  total  system. 

1.3  The  steps  undertaken  to  determine  the 

uncertainty  of  a  total  system's  performance 
are: 


3.  Analytic  Components 
of  the  Process 

3.1  Practical  Measurable 
Attributes 

3.2  Practical  Attribute 
Measures 

3.3  Performance 
Requirements,  Specific 

3.4  Performance 
Criteria,  Specific 

3-5  Measurement  Procedures 


(1)  Find  design  equations  that  relate  the 
subsystem  properties  to  the  total 
system  performance. 

(2)  Determine  the  subjective  probabilities 
for  the  subsystem  and  the  component 
properties  that  influence  the  total 
system  performance. 


4.  Planning  Components 
of  the  Process 

4.1  Analytic  Methods 

4.2  Parameter  Determinations 

4.3  Apparatus  for  Testing 

4.4  Personnel  for  Testing 

4.5  Test  Plans 

5.  Application  Components 
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5.1  Test  Execution 

5.2  Data  Analysis 

5.3  Findings  Interpretation 

5.4  Conclusions  and 
Recoemendations 


(3)  Utilize  the  Monte  Carlo  procedu-es  to 
generate  probability  distributions  for 
the  system  performance  characteristics. 

(*D  Compute  the  statistical  measures  of 
the  system  performance  probability 
distribution. 

(5)  Compare  the  statistical  measures  for 
the  different  time  periods  to  obtain 
indications  of  progress. 


6.  Further  Research  Areas 

6.1  Measurement  Systea 
Limitations 

6.2  Research  Potentials/ 
Prlorit'  es 

6.3  Research  Planning 
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The  performance  characteristics  critical  to 
the  aircraft's  mission  capability  are 
speed,  altitude,  range,  and  endurance. 


The  computational  methods  used  to  analyze 
the  data  were  the  measures  of  central 
tendency  (mean,  median,  mode)  and  the 
measures  of  dispersion  (range,  standard 
deviation,  variance). 
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This  report  describes  three  research 
approaches  to  the  problem  of  mathematically 
representing  human  performance  parameters 
in  weapon,  maintenance,  and  command  and 
control  systems.  In  the  first  approach, 
twenty  operations  research  analyses  and 
models  of  military  systems  were  examined  to 
determine  if  the  models  included  human 
factors  parameters  and  to  what  extent  they 
were  sensitive  to  variations  in  these 
parameters.  Although  many  of  the  functions 
of  the  systems  modeled  were  performed  by 
humans,  human  performance  parameters  were 
not,  in  general,  sufficiently  defined  to 
permit  mathematical  or  empirical  manipula¬ 
tion  within  a  man-machine  simulation  frame¬ 
work.  In  the  second  approach,  an  attempt 
was  made  to  establish  predictive  relation¬ 
ships,  based  on  regression  and  factor 
analysis  techniques,  between  human  engineer¬ 
ing  design  parameters  and  those  criteria  of 
systems  effectiveness,  such  as  maintenance 
task  time,  that  can  be  transformed  into  a 
more  molar  index  -  system  downtime.  The 
human  engineering  predictor-parameters 
accounted  for  50%  of  the  criterion  vari¬ 
ance.  In  the  third  approach,  a  series  of 
experiments  involving  real-time  simulation 
of  a  command  and  control  system  was  con¬ 
ducted  to  determine  if,  and  how,  a  computer 
might  aid  diagnostic  performance  (in  tacti¬ 
cal  decision  making)  in  threat  evaluations. 
The  system  output  or  criterion  of  effective¬ 
ness  was  the  degree  to  which  the  system 
assesses  the  true  state  of  threat.  With 
computer  aiding,  correct  decisions 
increased  by  13%. 
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Three  stages  can  be  identified  in  the 
evolution  of  human  engineering: 

(1)  Operationally-oriented  design  criteria 
development  (or  the  "knobs-and-dials" 
stage). 

(2)  Design  criteria  formalization  (publica 
tion  of  Human  Engineering  (HE) 
specifications,  standards,  handbooks, 
etc.). 

(3)  Systems  effectiveness  modeling  (or 
"systems-oriented  human  engineering"). 

Systems  effectiveness  modeling  is  concerned 
with  quantifying  the  impact  that  HE  design 
has  on  systems  parameters.  It  faces  the 
question:  if  one  maximizes  performance  for 
a  particular  tian-machine  interface,  what 
impact  will  this  have  on  overall  system 
performance? 

Three  major  categories  of  man-machine 
models  are  identified: 

(1)  Behavioristic  Models. 

(2)  Adaptations  of  Engineering  Models. 

(3)  Systems  Economic  Models. 

Behavioristic  models  derive  from  or 
parallel  the  stimulus  -  organismic  - 
response  (S-O-R)  paradigm  of  experimental 
psychology.  These  models  include: 
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6.1  Measureoent  System 
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6.2  Research  Potentials/ 
Priorities 

6.3  Research  Planning 
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(1)  A  "sensing  or  "input"  portion. 

(2)  A  "processing"  portion. 

(3)  An  "output  portion." 

Adaptations  of  engineering  models  basically 
form  two  classes:  Servotheoretic  Models 
and  Information  Theoretic  Models.  Systems 
Economic  Models  are  divided  into  two 
classes:  Operations  Research  Models  and 
Systems  Effectiveness  Models. 

2.6  The  major  parameters  of  system  effective¬ 
ness  have  been  defined  as  availability, 
capability  and  dependability.  Availability 
is  equivalent  to  the  system's  readiness  to 
perform  its  mission.  Dependability  is 
equivalent  to  a  measure  of  system  condition 
at  points  during  the  mission.  Capability 
is  a  measure  of  the  system's  ability  to 
achieve  the  mission  objectives.  These 
parameters  are  criteria  of  system  perform¬ 
ance  which  require  measurement  and  predic¬ 
tion.  A  paradigm  or  quantifiable  framework 
is  required  which  permits  assessment  of 
man's  performance  and  contribution  to  these 
criteria  -  parameters. 

3.5  The  following  are  goals  of  applied 

man-machine  models.  Such  models  should: 

(1)  Describe  and  quantify  the  functions  of 
both  man  and  machine. 

(2)  Allocate  functions  for  most  efficient 
utilization  of  man's  capabilities  and 
machine's  capabilities. 

(3)  Offer  design  alternatives  with 
functional  tradeoffs  between  design 
parameters  and  human  performance. 

(4)  Predict  influence  and  variations  in 
functional  considerations  on  systems 
criteria. 
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(5)  Be  capable  of  sensitivity  analysis. 

(6)  Be  capable  of  assimilating  new  human 
performance  data. 


(7)  Be  appropriate  to  the  level  of  systems 
description . 


(8)  Human  performance  measurement  should 
be  standardized  as  equivalent  to 
engineering  parameter  measures. 
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1.3  The  purpose  of  this  study  was  to  produce  a 
test  and  evaluation  guide  for  an  advance 
surface  ship  weapon  system.  This  guide  is 
intended  to  be  used  by  personnel  well 
qualified  operationally  and  technically, 
but  who  have  limited  test  and  evaluation 
experience. 
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The  guide  provides  an  overall  skeletal 
structure  of  events  that  take  place  in  T 
and  E  from  the  issuance  of  project 
assignment  to  composition  of  the  final 
report.  This  structure  provides  a 
step-by-step  procedure  that  aids  in 
organizing  the  total  effort. 
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Measures 
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of  the  Process 

4.1  Analytic  Methods 
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Standardized  formats  with  instructions 
provide  guidance  in  key  steps  such  as 
writing  the  test  plan  and  producing  final 
test  report.  Checklists  of  important 
actions  are  presented  and  reference  lists 
have  been  provided. 

The  structure  of  the  docunent  is  heavily 
influenced  by  the  process  the  test  team 
experiences. 

Numerous  examples  are  used  to  illustrate 
the  collection,  verification,  reduction, 
and  control  of  data.  A  standardized  test 
plan  format  is  presented  with  instructions 
for  use  and  follow-up.  A  standardized 
format  for  test  procedures  is  also 
supplied. 

The  use  of  simulators  for  physical  testing 
and  as  a  diagnostic  tool  is  discussed. 

The  processes  of  analysis  are  set  down  in  a 
step-by-step  procedure.  A  standardized 
test  report  format  is  presented  and  a 
glossary  is  provided. 
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This  report  discussed  the  Airborne  Warning 
and  Control  System  (AWACS). 

The  utility  of  AWACS  when  employed  as  an 
element  of  the  Tactical  Air  Control  System 
(TACS)  was  examined  and  nine  generic 
tactical  air  missions  were  selected  for 
this  evaluation. 

The  ultimate  requirement  of  the  AWACS  was 
to  kill  hostiles  and  destroy  targets. 

AWACS*  MOE's  were  divided  into  five 
distinct  categories:  (1)  Reaction  (receive 
a  mission,  request  a  message,  process  data, 
communicate  response);  (2)  Surveillance 
(detect,  identify  and  track  aircraft);  (3) 
Command  (allocation  of  available 
resources);  (4)  Control  (the  control  of 
friendly  aircraft)  and  (5)  Communications 
(on-board  communications  capability). 

Reaction  is  measured  in  units  of  time  and 
surveillance  ability  in  terms  of  numbers  of 
friendly/hostile  aircraft  detected, 
identified,  tracked,  etc.,  per  unit  of 
time.  Conmand  is  measured  by  the  ability  of 
AWACS  to  allocate  resources  in  terms  of 
number  and  percent  of  sorties  scrambled, 
immediate  response  requests  accomodated, 
and  sorties  diverted.  Control  reflects  the 
number  and  percent  of  friendly  aircraft 
under  control  per  unit  of  time  per  sortie. 
No  MOE  was  assigned  to  the  Communication 
category  -  -  this  function  is  implicit  in 
all  the  reaction  time  and  command  MOE's  and 
in  all  but  one  of  the  control  MOE's. 
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3.3  The  AWACS'  contribution  in  terms  of 
surveillance,  command,  control,  and 
communications  capabilities  was  the 
specific  area  of  interest. 

3.5  A  methodology  and  criteria  were  established 
for  assessing  the  system's  capability  using 
system-level  measures  of  effectiveness.  A 
set  of  MOE's  was  established  for  both 
levels  (the  AWACS  system  and  the  tactical 
mission).  To  obtain  a  standard  for 
comparison  the  scenerio  under  consideration 
was  analyzed  both  with  and  without  AWACS. 
The  incremental  differences  in  tactical 
mission  MOE's  combined  with  AWACS  system 
MOE's  provided  the  insight  into  the 
effectiveness  of  AWACS. 

5.4  It  was  felt  that  by  following  the 
methodology  developed  in  this  report,  the 
specific  contributions  of  the  AWACS  will  be 
clearly  measured  and  established.  Subse¬ 
quent  application  of  the  tactical  mission 
level  MOE's  will  yield  additional  insight 
into  the  operational  significance  of  the 
AWACS'  contribution  to  the  success  of  the 
tactical  mission. 

It  is  stated,  however,  that  no  single 
unique  MOE  can  be  generated  to  measure 
either  AWACS'  effectiveness  or  tactical 
mission  success. 
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1.1  The  systems  measurement  bed  is  a  means  of 
focusing  step-by-step  on  the  human  perform¬ 
ance  aspects  of  the  system  to  be  enhanced 
and  identifying  the  interrelationships  of 
the  human  factor  system  variables  in  order 
to  determine  productivity  under  varying 
conditions . 


2.  Contextual  Components 
of  the  Process 

2.1  System  Definition 

2.2  Mission  Definition 

2.3  Environment  Definition 
2. A  General  Constraints 

2.5  Performance 
Requirements,  Ultimate 

2.6  Performance 
Criteria,  Ultimate 


The  Job  of  leader  must  take  into  considera¬ 
tion  the  influence  of  the  situation  and 
styles  of  behavior  on  job  performance.  How 
a  leader  goes  about  carrying  out  the 
mission  objective  is  directed,  in  part,  by 
his  particular  style  of  behavior,  value 
system,  and  the  situation. 
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A  taxonomy  of  jobs  containing  cognitive 
variance  (responses  more  objectively 
characterized  as  right  or  wrong)  and 
noncognitive  variance  (responses  less 
objectively  characterized  as  desirable  or 
undesirable)  were  determined  from  this 
study.  The  system  measurement  bed  assists 
the  researcher  in  dealing  with  the 
different  measurement  characteristics  of 
the  two  classes  of  jobs. 

The  establishment  of  a  systems  measurement 
bed  calls  for  a  great  deal  of  subject 
expertise:  situations  have  to  be  designed, 
scenarios  written,  measurement  strategies 
devised,  and  computer  programs  prepared. 
Appropriate  experienced  personnel  must  be 
identified  to  serve  as  subjects.  Lastly, 
all  these  concepts,  materials,  and 
procedures  have  to  be  built  into  a 
logistically  feasible  space  where  relevant 
factors  and  criteria  can  be  incorporated. 
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This  paper  discusses  trends  in  the 
development  and  use  of  performance 
measures.  It  is  noted  that  Army  research 
to  develop  and  to  predict  criteria  of  human 
performance  has  attempted  to  achieve 
greater  relevance  between  performance 
measures  and  job  tasks. 

The  paper  discusses  the  various  performance 
criteria  utilized  in  the  past  to  predict 
individual  effectiveness;  these  criteria 
include  grades,  ratings,  and  performance 
tests.  It  describes  the  more  recently 
developed  measures  of  unit  effectiveness 
and  measures  dealing  with  human  factors 
problems  encountered  in  systems  analysis. 
Some  of  the  measures  mentioned  are  the 
"Skill  Qualification  Test"  (SQT) ,  devised 
as  a  performance  -  based  measure  in  one  of 
the  Army's  new  tactical  training  systems. 
This  method  provides  two-sided  free  play 
exercises  under  simulated  battlefield 
conditions  and  includes  a  specific  set  of 
operations  for  observing  and  recording 
actions  that  operational  personnel  have 
agreed  upon  as  relevant  to  mission  success. 

Other  Army  measurement  systems  described 
include  the  Organizational  Effectiveness 
(0E)  program,  using  a  Work  Environmental 
Questionnaire  (WEQ)  for  diagnostic 
purposes,  and  a  field  method  for  evaluating 
the  performance  of  Army  helicopter  pilots. 

The  paper  goes  on  to  describe  the  System 
Measurement  Bed  (SMB)  research  approach 
associated  with  the  concept  of  how  section 
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training  and  work  environment  interact  to 
influence  the  performance  of  the  individual 
on  the  job. 


The  paper  concludes  by  noting  that  there 
are  exciting  possibilities  in  the  future  of 
human  performance  systems  research  and  that 
analytic  techniques  in  this  area  have  the 
potential  to  impact  a  wide  range  of  areas 
beyond  the  man-machine  systems  in  the  Army. 
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1.3  This  study  is  concerned  with  the  collection 
and  comparison  of  measures  of  effectiveness 
(MOE's)  used  in  naval  studies  and  analyses. 

Volume  I  is  the  final  report  of  the  study 
and  includes  the  results  of  examining  213 
Navy  studies  of  system  effectiveness 
covering  virtually  all  aspects  of  naval 
warfare. 

The  results  of  this  study  'ire  presented  in 
several  different  forms.  First,  a  data 
base,  utilizing  two  types  of  formats  -  - 
Study  Review  Summary  or  MOE  Review,  was 
established  to  present  in  summary  form  the 
effectiveness  profile  of  each  study 
examined.  This  profile  presents  an  outline 
of  the  military  situation  addressed, 
variables  and  qualitative  factors 
considered,  and  the  special  assumptions  and 
limitations  in  MOE  formulation  and 
development.  In  addition,  a  general 
summary  of  measures  of  effectiveness  used 
in  naval  warfare  is  presented. 
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Of  the  studies  examined,  the  ASW  area 
accounted  for  371,  the  attack  area 
accounted  for  23X,  and  the  anti-air  warfare 
area  accounted  for  9S  of  the  warfare  areas 
considered.  The  remaining  31%  consisted  of 
mining  and  mine  countermeasures, 
surveillance,  strategic  systems,  electronic 
warfare,  amphibious  assault, 
communications,  command  and  control, 
navigation,  special  warfare, 
reconnaissance/intelligence,  logistics,  and 
ship  support. 
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Analysis  was  conducted  on  the  types  of 
variables  used  in  mathematical  formulation 
and  development.  The  results  showed  that 
nearly  of  all  independent  variables 
considered  were  associated  with  the 
particular  interests  of  the  author.  Only 
19%  of  all  independent  variables  related  to 
the  threat  or  target,  and  slightly  less 
than  3 %  related  to  the  physical  environment. 

In  Appendix  E  of  this  report  an  index  of 
the  measures  of  effectiveness  used  in  these 
studies  was  presented.  The  index  described 
the  system  and  its  function,  the  situation 
and  the  criterion  for  success,  and  lists 
the  measures  of  effectiveness  considered. 

Several  limitations  on  system  measurement 
in  these  studies  were  noted.  Briefly 
stated  they  are: 

(1)  The  criterion  for  success  is  seldom 
explicitly  stated. 

(2)  There  exists  more  than  one  way  of 
quantifying  how  well  the  criterion  for 
success  is  met. 

(3)  For  each  possible  mission  title  there 
is  more  than  one  way  of  defining  the 
mission. 

(4)  The  rationale  for  MOE  selection  is  not 
always  presented. 

(5)  Physical  environment  aspects  appear  to 
be  generally  ignored  or  casually 
treated  in  effectiveness  studies. 

(6)  It  appears  that  there  are  cases  where 
the  variable"  selected  for  model 
formulation  are  not  readily  (if  at 
all)  measureable  in  the  real  world. 

(7)  In  general,  the  MOE's  used  are  those 
that  are  readily  obtained  via  model 
development . 


A-428 


I  Topics  Relevant 

I  1 

1  1 

I  to  System  Development 

{Topic ! 

land  Evaluation  Technology 

:  no.  : 

ABSTRACT 

(8)  Very  seldom,  when  more  than  one  MOE  is 
identified,  is  a  ranking  of  importance 
performed  or  combined  measure 
developed  and  used. 

(9)  Expected  value  type  MOE's  are  most 
prevalent  in  force  level  studies 
whereas  probability  type  MOE's  are 
most  prevalent  in  subsystem  level 
studies . 

(10)  On  the  average,  over  twice  as  many 
independent  variables  occur  in  U.  ■* 
friendly  force  category  than  in  the 
threat  and  target  categories  combined. 

(11)  As  the  study  level  increases,  from 
subsystem  to  system  to  force  level, 
the  percentage  of  independent 
variables  in  the  friendly  force 
category  decreases  and  the  percentage 
of  independent  variables  in  the 
friendly  force  interaction  with  threat 
or  target  category  increases. 

(12)  It  is  not  easy  to  compare  similar 
effectiveness  studies. 
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1.1  This  document  contains  annotated 

bibliographies.  The  following  is  a  list  of 
the  10  specific  categories  or  headings  plus 
the  number  of  bibliographies  each  contains: 


(1)  Airborne  ASW  17 

(2)  Airborne  AAW  4 

(3)  Airborne  Attack  14 

(4)  Environmental  Systems  1 

(5)  Mining  2 

(6)  Mine  Counter  Measures  3 

(7)  Ocean  Surveillance  3 

(8)  Submarine  ASW  18 

(9)  Submarine  Attack  4 

(10)  Surface  ASW  15 


3.  Analytic  Components 
of  the  Process 

3.1  Practical  Measurable 
Attributes 

3.2  Practical  Attribute 
Measures 

3.3  Performance 
Requirements,  Specific 

3. A  Performance 

Criteria,  Specific 
3.5  Measurement  Procedures 

A.  Planning  Components 
of  the  Process 
A.1  Analytic  Methods 
A. 2  Parameter  Determinations 
A. 3  Apparatus  for  Testing 
A. A  Personnel  for  Testing 
A. 5  Test  Plans 

5.  Application  Components 
of  the  Process 

5.1  Test  Execution 

5.2  Data  Analysis 

5.3  Findings  Interpretation 
5. A  Conclusions  and 

Recommendations 


The  following  is  a  representative  sample  of 
missions: 

(1)  Submarine  search 

(2)  Sonobuoy  barrier  patrol 

(3)  Barrier  placement/patrol 

(4)  Surface  ship  defense 

(5)  Air  strike 

(6)  Close  air  support 

(7)  Surveillance  of  ocean  area 

(8)  Bathythermograph  maneuver 

(9)  Mine  clearance 

(10)  Contact  prosecution 

(11)  Search  and  destroy 

(12)  Submarine  attack  on  convoy 

(13)  Escort/screen 


6.  Further  Research  Areas 

6.1  Measurement  System 
Limitations 

6.2  Research  Potentials/ 
Priorities 

6.3  Research  Planning 
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3.2  The  following  is  a  representative  sample  of 
practical  attribute  measures: 

(1)  Probability  of  submarine  detection, 
localization,  and  kill. 

(2)  Ratio  of  the  incremental  improvement 
in  accomplishing  the  mission  to  the 
incremental  monetary  cost  of  such  an 
improvement . 

(3)  Detection  range  of  raid  relative  to 
the  vital  area  center  (CVA)  for  a 
given  intercept  range. 

Expected  number  of  targets  destroyed 
in  a  given  period  of  time. 

(5)  Difference  in  fuel  consumption  due  to 
the  bathythermograph  maneuver. 

(6)  Total  force  level  required  to  clear  a 
given  area  in  a  given  time. 

(7)  Expected  number  of  ships  hit. 

(8)  Elapsed  time  to  target  detection. 

(9)  Maximum  exposure  time  of  the  submarine. 

3.3  The  following  is  a  representative  sample  of 

the  criteria  for  success: 

(1)  Detection  of  submarine. 

(2)  Suppression  of  submarine  activity. 

(3)  Destruction  of  target. 

(H)  Successful  attack  capability. 

(5)  Low  cost  measurement  of  the  vertical 
ocean  temperature  profile. 

(6)  Survival  of  aircraft  and  planting  of 
mines. 
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(7)  Clearance  of  minefield 

(8)  Surveillance  and  establishment  of  the 
track  of  ships  at  sea. 


(9)  Preparation  for  attack  in  the  least 
possible  time  without  being 
counter-detected . 


(10)  Insurance  of  the  safe  passage  of 

convoys,  strike  groups,  and  amphibious 
forces  in  the  presence  of  hostile 
submarines. 
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The  U.S.  Army  must  learn  to  fight 

outnumbered  and  win.  The  priorities  to 

these  goals  include  the  following: 

(1)  Suppressive  fires  against  enemy  air 
defense  systems,  especially 
radar-directed  systems,  to  permit 
scout  and  attack  helicopters  to 
operate  more  effectively. 

(2)  Attack  and  counterattack  on  reverse 
slopes  to  protect  attacking  forces 
from  long  range  enemy  observation  and 
fire. 

(3)  Operating  in  darkness  or  other 
conditions  of  reduced  visibility  to 
reduce  range  and  accuracy  of  enemy 
observation  and  fire. 

(4)  Precision,  discipline,  3peed,  and 
security  in  the  directing  and 
reporting  of  the  battle  in  order  to 
win  the  battle  quickly,  unimpeded  by 
enemy  countermeasures. 

(5)  Adequacy  of  stowed  loads  of 
ammunition  and  fuel;  speed, 
responsiveness,  and  security  of 
resupply  systems  to  reduce  the  need  to 
resupply,  but  to  ensure  prompt 
resupply  when  it  is  needed. 

(6)  Detection  and  identification  of  the 
enemy  at  maximum  possible  distances 
from  the  friendly  main  body  to  prevent 
engagement  of  the  main  body  under 
adverse  conditions — when  it  is 
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unwarned,  poorly  deployed,  not  poised 
to  fight. 

(7)  Firing  fast  first  in  a  tank-antitank 
battle,  especially  at  targets  that  can 
shoot  back.  In  a  tank  duel,  accuracy 
is  important — firing  first  is  more 
important. 

(8)  Control  and  distribution  of  tank  anti¬ 
tank  fires  to  kill  targets  rapidly  and 
save  ammunition  for  engaging  the  next 
attacking  echelon. 

(9)  Battlefield  movement  only  along 
covered  avenues-making  maximum  use  of 
terrain  to  evade  enemy  long  range 
observation  and  fire. 

(10)  Suppressive  fires  delivered  from 
overwatching  positions  to  reduce  the 
chance  that  maneuvering  friendly 
forces  can  be  seen  and  t  igaged  by  the 
enemy. 

2.3  The  assault  takes  place  from  the  last 
available  cover  and  carries  onto  the 
objective.  The  assault  is  normally 
initiated  from  a  wedge;  however,  terrain 
and  individual  tank  actions  will  govern  the 
actual  alignment. 
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Requirements,  Ultimate 
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Criteria,  Ultimate 


This  handbook  was  an  exposition  of  proven 
methods  and  materials  for  the  engineering 
design  of  the  recoilless  rifle  weapon 
systems. 

The  purpose  of  this  handbook  was  to  guide 
the  engineer,  the  mature  practitioner  as 
well  as  the  novice,  directly  to  project 
goals.  It  provided  a  comprehensive  summary 
of  the  available  relevant  technology  and 
the  system  engineering  rationale. 

The  basic  input  requirement  to  the  weapon 
system  was  the  kill  probability  for  a 
particular  target  and  specified  range. 


3.  Analytic  Components 
of  the  Process 

3.1  Practical  Measurable 
Attributes 

3-2  Practical  Attribute 
Measures 

3.3  Performance 

Requirements,  Specific 
3-4  Performance 

Criteria,  Specific 
3-5  Measurement  Procedures 

4.  Planning  Components 

of  the  Process  3.2 

4.1  Analytic  Methods 

4.2  Parameter  Determinations 

4.3  Apparatus  for  Testing 

4.4  Personnel  for  Testing 

4.5  Test  Plans 

5.  Application  Components 
of  the  Process 

5.1  Test  Execution 

5.2  Data  Analysis 

5.3  Findings  Interpretation 
5-4  Conclusions  and 

Recommendations 

6.  Further  Research  Areas 

6.1  Measurement  System 
Limitations 

6.2  Research  Potentials/ 
Priorities 

6.3  Research  Planning 


The  kill  probability,  in  turn,  places 
requirements  on  the  hit  probability  and 
fire  power  of  the  weapon  being  designed. 

As  these  requirements  are  traced  further 
through  the  system,  it  is  found  that  all 
components  of  the  rifle  are  affected.  The 
result  of  this  interaction  is  a  system 
weight  for  a  given  terminal  ballistic 
requirement. 

Kill  probability  was  defined  as  the  product 
of  hit  probability  of  a  kill  given  a  hit. 
From  the  definition  of  target  vulnerable 
area,  conditional  kill  probability  can  be 
expressed  as  the  ratio  of  the  vulnerable 
area  to  the  presented  area. 

Hit  probability  was  defined  as  the 
probability  of  a  hit  or  hits  on  a  target 
occurring  out  of  a  given  number  of  rounds 
fired  at  a  target.  For  a  specified  target 
and  weapon  system,  the  hit  probability 
depends  only  on  the  overall  weapon 
dispersion.  The  principal  sources  of  these 
dispersions  or  firing  errors  are  range 
estimation  aiming,  muzzle  velocity 
variation,  system  jump  and  cant,  crosswind 
and  the  fire  control  equipment. 
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1.3  In  the  CE-75  man-machine  interface  investi¬ 
gation,  the  following  procedures  were  under¬ 
taken: 

(1)  Creation  of  a  data  base 

(2)  Analysis  of  systems  concepts 


Contextual  Components 

of  the  Process 

(3) 

2.1 

System  Definition 

2.2 

Mission  Definition 

(4) 

2.3 

Environment  Definition 

2.4 

General  Constraints 
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Performance 

Requirements,  Ultimate 

(5) 

2.6 

Performance 

(6) 

Criteria,  Ultimate 

Expansion  of  a  data  base 
Examination  of  suitable  mode  ,s 
Review  of  the  personnel  subsystem 
Critical  incident  analysis 


3.  Analytic  Components 
of  the  Process 

3.1  Practical  Measurable 
Attributes 

3.2  Practical  Attribute 
Measures 

3.3  Performance 
Requirements,  Specific 

3  -  **  Performance 

Criteria,  Specific 
3.5  Measurement  Procedures 

1).  Planning  Components 
of  the  Process 

4.1  Analytic  Methods 

4.2  Parameter  Determinations 

4.3  Apparatus  for  Testing 

4.4  Personnel  for  Testing 
*1.5  Test  Plans 


(7)  Measurement  of  man's  contribution  to 
system  effectiveness 

A  system  definition  is  a  conceptual  frame¬ 
work  for  attacking  problems.  In  its  broad¬ 
est  terms  a  system  was  comprised  of  hard¬ 
ware,  facilities,  logistic  support,  and  the 
trained  manpower  required  for  operation  in 
a  particular  environment.  For  CE-75,  the 
MMI  system  included  the  entire  collection 
of  men,  facilities,  and  equipment  in  the 
CE-75  Field  Army  Tactical  Communications 
System.  This  system  also  included  the  area 
and  command  systems. 


S-  Application  Components 
of  the  Process 

5.1  Test  Execution 

5.2  Data  Analysis 

5.3  Findings  Interpretation 

5.4  Conclusions  and 
Recommendations 


6.  Further  Research  Areas 

6.1  Measurement  System 
Limitations 

6.2  Research  Potentials/ 
Priorities 

6.3  Research  Planning 
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2.2  In  a  broad  sense,  the  mission  of  the  CE-75 
communications  system  was  to  provide  a 
means  for  the  timely  transfer  of  meaningful 
and  significant  information  from  action 
officer  to  action  officer.  In  a  more 
specific  sense,  what  was  considered  were 
the  estimates  of  how  much  time  can  be 
allowed  for  the  information  transfer 
process,  message  priorities,  and  a  delinea¬ 
tion  of  the  organization  to  be  served. 
Additional  elements  of  the  mission  were: 

(1)  How  much  information  and  of  what  kind 
need  be  transferred  between 
individuals? 

(2)  What  mode  should  be  used  for 
transmission  of  this  information? 

2.4  A  man-machine  interface  is  the  boundary  at 
which  a  man  and  a  machine  interface  in 
order  to  achieve  a  system  objective.  The 
extent  of  this  boundary  was  constrained  by 
three  factors: 

(1)  Tasks  required  of  both  the  man  and  the 
machine  to  attain  the  system 
objective. 

(2)  Capabilities  and  limitations  of  the 
machine. 

(3)  System  objectives  as  affected  by 
environment,  personnel  policies,  and 
equipment  use. 
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This  conference  provided  a  forum  for 
discussion  of  the  question:  "Is  standardi¬ 
zation  necessary  in  human  factors?"  The 
papers  presented,  although  they  did  not 
completely  answer  this  question,  did  give 
some  guidance  to  research  groups  regarding 
the  necessity  for  insuring  that  human 
factors  research  is  mission  oriented  and 
relevant  to  the  needs  of  its  sponsors. 

There  was  considerable  discussion  with 
regard  to  a  proposal  that  a  data  bank  be 
established  for  use  by  human  factors  re¬ 
searchers.  It  was  pointed  out  that  a  data 
bank  could  supply  copies  of  specific 
tables,  figures  and  formulae  directly  to  the 
requester  which  would  reduce  the  amount  of 
duplicated  effort  presently  expended  by 
researchers.  Some  expressed  the  fear  that 
such  a  system  would  seduce  the  user  into 
weighting  poorly  collected,  unreliable  data 
equally  with  carefully  generated  reliable 
data.  The  rebuttal  to  this  point  was  that 
experienced  investigators  could  generally 
evaluate  a  set  of  data  in  terms  of  its 
"reasonableness"  and  source.  It  was  the 
consensus  that  the  data  bank  concept  could 
be  of  great  value  although  there  was  some 
concern  for  the  potential  misuse  of  such  a 
system. 


5.  Application  Components 
of  the  Process 

5.1  Test  Execution 

5.2  Data  Analysis 

5.3  Findings  Interpretation 

5.4  Conclusions  and 
Recommendations 

6.  Further  Research  Areas 

6.1  Measurement  System 
Limitations 

6.2  Research  Potentials/ 
Priorities 

6.3  Research  Planning 


It  was  noted  that  the  value  of  a  data  re¬ 
trieval  system  would  be  enhanced  if  one 
could  query  the  system  about  performance  on 
specified  tasks  under  specified  conditions. 
However,  there  was  some  reservation  with 
regard  to  the  practicality  of  standardizing 
tasks  and/or  variables.  The  comment  was 
made  that  tasks  might  be  defined  in  terms  of 
human  functioning  compartmentalized  by  the 
muscle  groups  involved  rather  than  apparatus 
used. 
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Another  point  of  view  was  presented  to  the 
effect  that  standardization  of  tasks  may 
be  less  -mportant  than  specification  of 
conditions  under  which  the  performance  data 
is  collected.  The  major  concern  throughout 
the  discussion  was  for  end-goal,  or  mission 
completion,  requirements  in  research.  This 
was  considered  by  most  participants  to  be 
more  important  than  standardization. 

There  was  lengthy  discussion  of  work  measure¬ 
ment  problems  and  considerable  dissatisfac¬ 
tion  was  expressed  with  currently  available 
measures  of  human  work. 

There  was  also  discussion  of  field  experi¬ 
mentation  versus  laboratory  experimentation 
and  the  need  for  research  on  mission-oriented 
tasks.  It  was  stressed  that  the  operational 
environment  must  be  kept  in  mind  when  planning 
human  factors  experiments.  The  need  for  a 
systems  approach  which  would  include  the 
study  of  social  and  physical  environments 
in  mission-oriented  situations  was  also  noted. 
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This  volume  summarizes  progress  made  in  the 
area  of  small  arms  weapon  system  test 
methodology. 


Contextual  Components 
of  the  Process 

2.1  System  Definition 

2.2  Mission  Definition 

2.3  Environment  Definition 

2.4  General  Constraints 

2.5  Performance 
Requirements,  Ultimate 

2.6  Performance 
Criteria,  Ultimate 


In  this  phase  of  the  study  test  facilities 
were  constructed  for  the  evaluation  of 
small  arms  weapons  utilized  by  the 
infantry. 


The  object  of  the  study  was  to  simulate  as 
nearly  as  possible  actual  combat 
conditions. 


Analytic  Components 
of  the  Process 

3.1  Practical  Measurable 
Attributes 

3.2  Practical  Attribute 
Measures 

3.3  Performance 
Requirements,  Specific 

3.A  Performance 

Criteria,  Specific 
3.5  Measurement  Procedures 


Several  categories  of  MOE's  were  estab¬ 
lished:  accuracy,  responsiveness,  sustaina¬ 
bility,  reliability,  portability,  and  com¬ 
patibility,  and  signature  effects.  These 
characteristics  are  measured  under  attack 
and  quickfire  and  defense  modes. 


Quantitative  data  were  used  to  answer 
questions  concerning  the  weapons  system 
performance. 


Planning  Components 
of  the  Process 

4.1  Analytic  Methods 

4.2  Parameter  Determinations 

4.3  Apparatus  for  Testing 

4.4  Personnel  for  Testing 

4.5  Test  Plans 


Instrumentation  was  developed  and  data 
processing  and  computer  equipment  installed 
at  the  test  facilities.  The  goal  was  to 
use  more  than  one  test  soldier  at  a  time 
and  yet  be  able  to  measure  individual 
performance. 


Application  Components 
of  the  Process 

5.1  Test  Execution 

5.2  Data  Analysis 

5.3  Findings  Interpretation 

5.4  Conclusions  and 
Recommendations 


Further  Research  Areas 

6.1  Measurement  System 
Limitations 

6.2  Research  Potentials/ 
Priorities 

6.3  Research  Planning 


A-442 


i  Topics  Relevant 

1  1 

1  1 

i  to  System  Development 

I  Topic', 

land  Evaluation  Technology 

1  No.  i 

ABSTRACT 

4.1  Multivariate  analysis  is  us-.d  to  measure 

more  than  one  measure  of  effectiveness,  and 
takes  into  consideration  the  variability  of 
sample  size  and  the  interrelationships 
among  the  various  measures.  It  permits  the 
observation  of  statistical  weighting 
factors  and  is  considered  the  key  to  the 
operational  weighting  of  factors  which 
influence  the  combat  environment. 

Regression  analysis  using  miss  distance 
data  was  used  for  comparison  of  different 
types  of  rifles. 

4.5  Three  facilities  are  operational:  attack, 
defense,  and  quickfire.  In  this  report,  no 
specific  test  plans  were  presented. 

5.3  This  report  discusses  in  general  terms  the 
operational  capabilities  of  the  test 
facilities.  Data  are  presented  in  various 
appendices  (not  in  our  files). 
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1.3  This  report  describes  a  methodology  for 
evaluating  system  peculiar  Test, 
Measurement,  and  Diagnostic  Equipment 
(TMDE),  including  physical  and  operational 
characteristics.  Procedures  are  provided 
for  initial  inspection,  physical 
characteristics,  safety,  performance, 
extreme  environments,  r-f  interference, 
reliability,  maintenance,  and  human 
factors.  Supplementary  instructions  are 
provided  for  identifying  the  test  item, 
documenting  test  criteria,  developing 
performance  tests,  environmental  tests, 
test  plan  organization,  and  maintenance 
evaluation  of  the  TMDE. 
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5.2  Data  Analysis 

5-3  Findings  Interpretation 
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1.  State  of  the  Art  Review 
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2.1  The  MICV  (Mechanized  Infantry  Combat 

Vehicle)  was  a  lightly  armored  vehicle 
designed  to  provide  rapid  cross-country 
mobility,  large  volumes  of  fire  power  from 
orf  nic  weapons,  armored  protection  to  a 
'  'l  r  equipped  rifle  squad,  and  communi- 
cat  ons  between  all  elements  of  the  unit. 
It  was  designed  to  enable  the  rifle  squad 
to  fight  effectively  in  both  mounted  and 
dismounted  roles  in  offensive,  defensive, 
or  retrograde  formations. 

2.3  The  tests  on  the  MICV  were  performed  under 
climatic  conditions  best  described  as 
European  winter  thaw. 
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of  the  Process 

3.1  Practical  MeasuraDle 
Attributes 

3.2  Practical  Attribute 
Measures 

3.3  Performance 
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3.5  Measurement  Procedures 


During  a  specific  phase  (i.e.,  fording,  in 
which  is  measured  vehicle  egress  and 
ingress  ability  at  water  crossings),  time 
constraints  did  not  permit  waiting  for  the 
desired  winter  conditions.  The  intent  was 
to  specifically  address  wetness  effect  on 
system  functions  under  winter  thaw 
conditions. 
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5.3  Findings  Interpretation 

5.4  Conclusions  and 
Recommendations 


People  who  participated  in  the  present 
study  included  six  scout  drivers,  six  tract 
commanders,  nine  tank  drivers,  and  nine 
tank  commanders.  Infantry  players  received 
both  individual  and  squad  training  in  their 
duties  in  the  MICV.  The  gunner  training 
developed  the  gunners  to  perform  at  a 
predetermined  safe  level  of  proficiency, 
though  not  at  the  level  to  perform  combat 
duty. 


6.  Further  Research  Areas 

6.1  Measurement  System 
Limitations 

6.2  Research  Potentials/ 
Priorities 

6.3  Research  Planning 
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Human  factors  engineering  (HFE)  is  the  ap¬ 
plication  of  scientific  principles  concern¬ 
ing  human  physical  and  psychological  charac 
teristics  to  the  design  of  equipment,  there 
by  increasing  speed  and  precision  of  opera¬ 
tions  providing  maximum  maintenance  effi¬ 
ciency,  reducing  fatigue,  and  simplifying 
operations.  HFE  requires  the  consideration 
of  human  characteristics  such  as  separate 
anthropometries,  intellectual  abilities, 
sensory  capacities,  mobility,  muscle 
strength,  basic  skills,  and  the  capacity  to 
learn  new  skills. 

The  study  was  conducted  under  a  simulated 
battle  environment  with  the  physical  and 
environmental  conditions  duplicating  those 
to  be  found  in  the  equipment's  future  use. 

Battlefield  Mobility  -  Within  the 

limitations  imposed  by  the  nature  of  the 
equipment,  physical  design  should  provide 
integral  features  and/or  special  provisions 
to  facilitate  lifting  and  carrying  by  the 
individual  soldier  with  minimum  loss  of 
efficiency.  During  conduct  of  common  MTP 
3-3-502,  battlefield  mobility,  measurements 
and  observations  will  be  made  to  determine 
HFE  design  characteristics  which  enhance  or 
unduly  limit  the  man-portability  and 
man-transportability  of  the  equipment. 

Some  subjects  for  HFE  consideration  are 
listed  below.  These  will  be  explored  as 
appropriate  for  the  particular  test  item: 
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(1)  Weight,  dimensions,  and  configurations 
with  respect  to  anthropometric  charac¬ 
teristics  of  representative  test 
soldiers. 

(2)  Method  of  carry  and  means  to  secure 
during  marches. 

(3>  Location,  design,  and  texture  of 

gripping  areas  and  carrying  handles. 

(4)  Location,  configuration,  and  surface 
texture  of  areas  which,  although  not 
specifically  intended  for  the  purpose, 
may  be  used  for  lifting  and  carrying 
heavy  items. 

(5)  Mutual  interference  with  individual 
equipment  and  clothing. 

(6)  Human  limitations  to  endure  strain, 
fatigue,  and  discomfort  while  carrying. 

(7)  Effect  on  combat  readiness  of  the 
soldier. 

(8)  Handling  characteristics  during  combat 
movements  which  require  the  soldier  to 
run,  jump,  hit  the  ground  rapidly, 
roll,  and  assume  various  firing 
positions. 

(9)  Capabilities  and  limitations  for 
carrying  over  all  types  of  terrain. 

(10)  Ease  of  carry  and  freedom  from  inter¬ 
ference  while  carrying  in  air  and 
ground  vehicles. 

(11)  Ease  of  carry  and  delivery  by 
individual  parachutists. 

(12)  Human  capabilities  and  limitations 
with  respect  to  distance  of  carry. 

(13)  Weight,  configuration,  center  of 
balance,  and  load  distribution  of 
items  which  are  transported  by  two  or 
more  men. 
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(14)  Compatibility  of  design  for  use  with 
standard  individual  load  carrying 
equipment. 

(15)  Packaging,  protection,  or  other 
preparation  required  to  ready  the  item 
for  field  transport. 

Photographs,  motion  pictures,  videotapes, 
and  fast  frame  photographs  were  used  as 
aids  evaluating  human  engineering  aspects 
of  the  service  test. 


U.S.  Army  Test  and  Evaluation  Command.  Materiel  test  procedure  6-2-50?. 
(Electronics  Proving  Ground).  Fort  Benning,  GA:  U.S.  Army  Infantry  Board, 
August  1969.  (AD-720  976). 


i  Topics  Relevant 

1  1 

1  1 

1  to  System  Development 

i Topic! 

!  Ho.  i 

ABSTRACT 

1.  St»te  of  the  Art  Review 
of  the  Process 

1 . 1  Cenertl  System 
Measurements 

1.2  System  Taxonomy 
Model  (STM) 

1.3  Overall  Conceptual 
Process  Model  (CPM) 

2.  Contextual  Components 
of  the  Process 

2.1  System  Definition 

2.2  Mission  Definition 

2.3  Environment  Definition 

2.4  General  Constraints 

2.5  Performance 
Requirements,  Ultimate 

2.6  Performance 
Criteria,  Ultimate 


Human  engineering  is  defined  as  the  design 
of  equpment,  man-machine  systems,  and  human 
tasks  for  the  most  effective  human  accom¬ 
plishment  of  the  job.  Such  engineering 
requires  consideration  of  human  characteris¬ 
tics  such  as  anthropometries,  intellectual 
abilities,  sensory  capacities,  mobility, 
muscle  strength,  basic  skills,  and  the 
capacity  for  learning  new  skills.  In 
military  human  factors  engineering,  the 
designer  must  consider  human  limitations 
imposed  by  the  environmental  conditions 
typical  of  military  situations  of  use  where 
the  operator  is  often  working  under  stress 
and  fatigue. 


3.  Analytic  Components 

of  the  Process 

3.1  Practical  Measurable 
Attributes 

3.2  Practical  Attribute 
Measures 

3.3  Performance 
Requirements,  Specific 

3.4  Performance 
Criteria,  Specific 

3-5  Measurement  Procedures 

4.  Planning  Components 

of  the  Process 

4.1  Analytic  Methods 

4.2  Parameter  Determinations 

4.3  Apparatus  for  Testing 

4.4  Personnel  for  Testing 

4.5  Test  Plans 

5.  Application  Components 

of  the  Process 

5.1  Test  Execution 

5.2  Data  Analysis 

5.3  Findings  Interpretation 

5.4  Conclusions  and 
Recoenendatlons 

6.  Further  Research  Areas 

6.1  Measurement  System 
Limitations 

6.2  Research  Potentials/ 
Priorities 

6.3  Research  Planning 


The  subtests  given  in  this  Materiel  Test 
Procedure  (MTP)  are  performed  on  a 
selective  basis  as  required  for  a  specific 
item  of  equipment.  The  eight  subtests  and 
their  objectives  are  as  follows: 

(1)  Control-Display  Relationships  -  The 
objective  of  this  subtest  is  to 
determine  the  degree  to  which  the  test 
item  design  contributes  to  ease  of 
operation  through  incorporation  of 
preferred  display  and  associated 
control  location  relative  to  each 
other  and  to  operational 
characteristics. 

(2)  Visual  Displays  -  The  objective  of 
this  subtest  is  to  determine  the 
suitability  of  visual  displays 
relative  to  type,  size,  location, 
readability,  consistency,  and 
operational  characteristics. 
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(3)  Auditory  Warning  Devices  -  The 
objective  of  this  subtest  is  to 
determine  the  suitability  of  auditory 
warning  devices  relative  to  human 
factors  aspects  and  operational 
characteristics. 

(4)  Controls  -  The  objective  of  this 
subtest  is  to  determine  the 
suitability  of  controls  relative  to 
type,  size,  application,  location, 
coding,  consistency,  and  operational 
characteristics. 

(5)  Labeling  -  The  objective  of  this 
subt-est  is  to  determine  the  suitabil¬ 
ity,  readability,  and  consistency  of 
labeling  used  for  critical  markings, 
identification,  and  instructions. 

(6)  Workspace  Design  and  Layout  -  The 
objective  of  this  subtest  is  to 
determine  the  suitability  of  workspace 
relative  to  location,  size, 
accessibility,  and  configuration. 

(7)  Operator  Comfort  and  Lack  of  Inter¬ 
ference  -  The  objective  of  this  sub¬ 
test  is  to  determine  if  operator  com¬ 
fort  and  lack  of  interference  aspects 
are  satisfactory. 

(8)  Special  Observational  Tests  -  The 
objective  of  this  subtest  is  to 
determine  the  cause  of  special  human 
factors  engineering  problem  areas 
noted  during  some  phase  of  equipment 
operations  or  testing. 
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U.S.  Army  Test  and  Evaluation  Connand.  Materiel  test  procedure  10-2-505 
(General  equipment  test  activity).  Fort  Benning,  GA:  U.S.  Army  Infantry 
Board,  September  1971.  (AD-729  855). 


!  Topics  Relevant 
I  to  System  Development 
land  Evaluation  Technology 

1.  State  of  the  Art  Review 

of  the  Process 

1 . 1  General  System 
Measurements 

1.2  System  Taxonomy 
Model  (STM) 

1.3  Overall  Conceptual 
Process  Model  (CPM) 

2.  Contextual  Components 

of  the  Process 

2.1  System  Definition 

2.2  Mission  Definition 

2.3  Environment  Definition 

2.4  General  Constraints 

2.5  Performance 
Requirements,  Ultimate 

2.6  Performance 
Criteria,  Ultimate 

3.  Analytic  Components 

of  the  Process 

3.1  Practical  Measurable 
Attributes 

3.2  Practical  Attribute 
Measures 

3.3  Performance 
Requirements,  Specific 

3.4  Performance 
Criteria,  Specific 

3.5  Measurement  Procedures 


Planning  Components 
of  the  Process 

4.1  Analytic  Methods 

4.2  Parameter  Determinations 

4.3  Apparatus  for  Testing 

4.4  Personnel  for  Testing 

4.5  Test  Plans 

Application  Coeiponents 
of  the  Process 

5.1  Test  Execution 

5.2  Data  Analysis 

5-3  Findings  Interpretation 
5.4  Conclusions  and 
Recommendations 

Further  Research  Areas 

6.1  Measurement  System 
Limitations 

6.2  Research  Potentials/ 
Priorities 

6.3  Research  Planning 


I  Topic 
!  No. 


_ ABSTRACT _ 

Experience  has  shown  that  the  incorporation 
of  human  engineering  design  criteria, 
principles,  and  practices  has  improved 
mission  success  through  the  integration  of 
the  human  into  the  system,  subsystem, 
equipment,  or  facility.  Furthermore,  the 
cost  effectiveness  ratio  has  generally 
improved  as  a  result  of  the  application  of 
optimum  man-item  design  through  increased 
simplicity,  improved  safety  of  operations, 
and  reduced  training  and  maintenance 
requirements. 

The  attributes  that  were  measured  in  this 
study  were  noise,  visibility,  thermal 
considerations,  pressure  considerations, 
ventilation,  vibration,  and  radiation. 

The  equipment  that  was  required  in  this 
study  is  as  follows: 

(1)  Linear  measuring  devices,  tape 
measures,  rules,  etc. 

(2)  Scales,  balances,  and/or  other 
weighing  devices. 

(3)  Temperature  sensors. 

(4)  Pressure  Sensors. 

(5)  Octave-band  filter  set. 

(6)  General  purpose  sound  level  meters. 

(7)  Light  meters. 

(8)  Vibrometer. 
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(9)  Radiation  measuring  equipment  (as 
applicable) . 

(10)  Photographic  equipiment. 

(11)  Gas  sample  set. 


(12)  Accoustical  chamber  with  sound 
pressure  apparatus. 


U.S.  Department  of  the  Army.  Army  training  and  evaluation  program  for 
mechanized  Infantry  battalion  and  combined  arms  task  force  (ARTEP  7-45) 
Washington,  DC:  Headquarters,  Dept  of  the  Army,  September  1975. 


ABSTRACT 


This  ARTEP  serves  as  a  basis  for  developing 
a  mechanized  infantry  battalion's  training 
and  evaluation  program.  The  program  is 
designed: 


I  Topics  Relevant 
I  to  System  Development  I  Topic 
land  Evaluation  Technology  j  No. 

1.  State  of  the  Art  Review  1.1 

of  the  Process 

1.1  General  System 
Measurements 

1.2  System  Taxonomy 
Model  (STM) 

1.3  Overall  Conceptual  (1) 

Process  Model  (CPM) 

2.  Contextual  Components 
of  the  Process 

2.1  System  Definition 

2.2  Mission  Definition 

2.3  Environment  Definition 

2.0  Ceneral  Constraints  (2) 

2.5  Performance 
Requirements,  Ultimate 

2.6  Performance 
Criteria,  Ultimate 

3.  Analytic  Components 

of  the  Process  (3) 

3.1  Practical  Measurable 
Attributes 

3.2  Practical  Attribute 
Measures 

3-3  Performance 

Requirements,  Specific 
3.0  Performance  (mn 

Criteria,  Specific 
3.5  Measurement  Procedures 

0.  Planning  Components 
of  the  Process 
0.1  Analytic  Methods 
0.2  Parameter  Determinations 
0.3  Apparatus  for  Testing 
0.0  Personnel  for  Testing 
0.5  Test  Plans 


To  evaluate  the  ability  of  the 
battalion  to  serve  as  the  nucleus  of  a 
combined  arms  task  force  performing 
specified  missions  under  simulated 
combat  conditions. 

To  provide  a  guide  of  training 
objectives  by  specifying  minimum 
standards  of  performance  for 
combat-critical  missions  and  tasks. 

To  evaluate  the  efficiency  and 
effectiveness  of  past  training  of  all 
echelons  of  the  battalion  from 
crew/ squad  through  battalion/task 
force. 

To  provide  an  assessment  of  future 
training  needs. 


5.  Application  Components 
of  the  Process 

5.1  Test  Execution 

5.2  Data  Analysis 

5.3  Findings  Interpretation 
5. A  Conclusions  and 

Recommendations 

6.  Further  Research  Areas 

6.1  Measurement  System 
Limitations 

6.2  Research  Potentials/ 
Priorities 

6.3  Research  Planning 
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U.S.  Department  of  the  Army.  Skill  Qualification  Test  notice  for  MQS  11B  SQT 
3  Infantryman  (No.  7-11B3-N).  Washington,  DC:  Headquarters,  Dept,  of  the 
Army,  June  1976.  (Pamphlet). 


Topics  Relevant 

1  1 
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{Topic! 
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:  no.  ; 

ABSTRACT 

1 .  State  of  the  Art  Review 
of  the  Process 

1.1  General  System 
Measurements 

1.2  System  Taxonomy 
Model  (STM) 

1.3  Overall  Conceptual 
Process  Model  (CPM) 

2.  Contextual  Components 
of  the  Process 

2.1  System  Definition 

2.2  Mission  Definition 

2.3  Environment  Definition 
2. A  General  Constraints 

2.5  Performance 
Requirements,  Ultimate 

2.6  Performance 
Criteria,  Ultimate 


1.1  The  Skill  Qualification  Test  (SQT)  tests  a 
soldier  s  ability  to  do  those  tasks  in  his 
Military  Occupational  Specialty  (MOS)  that 
are  most  important  to  his  survival  in 
combat,  to  the  accomplishment  of  his  job, 
and  to  his  unit’s  mission.  The  SQT  may 
contain  up  to  three  parts:  the  written 
component,  the  hands-on  component,  and  the 
performance  certification  component. 

3.1  A  soldier's  ability  to  perform  a  task  is 
measured  by  the  SQT. 


3.  Analytic  Components 
of  the  Process 

3.1  Practical  Measurable 
Attributes 

3.2  Practical  Attribute 
Measures 

3.3  Performance 
Requirements,  Specific 

3.  A  Performance 

Criteria,  Specific 
3.5  Measurement  Procedures 


A.  Planning  Components 
of  the  Process 
A. 1  Analytic  Methods 
A. 2  Parameter  Determinations 
A. 3  Apparatus  for  Testing 
A. A  Personnel  for  Testing 
A. 5  Test  Plans 


5.  Application  Components 
of  the  Process 

5.1  Test  Execution 

5.2  Data  Analysis 

5.3  Findings  Interpretation 
5. A  Conclusions  and 

Recommendations 


6.  Further  Research  Areas 

6.1  Measurement  System 
Limitations 

6.2  Research  Potentials/ 
Pr4 jrities 

6.3  Research  Planning 
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U.S.  Department  of  the  Army.  Tank  gunnery  (Field  Manual  17-12).  Washington 
DC:  Headquarters,  Dept,  of  the  Army,  March  1977. 


ABSTRACT 


1.  State  of  the  Art  Review 

of  the  Process 

1.1  General  System 
Measurements 

1.2  System  Taxonomy 
Model  (STM) 

1.3  Overall  Conceptual 
Process  Model  (CPM) 

2.  Contextual  Components 

of  the  Process 

2.1  System  Definition 

2.2  Mission  Definition 

2.3  Environment  Definition 

2.4  General  Constraints 

2.5  Performance 
Requirements,  Ultimate 

2.6  Performance 
Criteria,  Ultimate 

3.  Analytic  Components 

of  the  Process 

3.1  Practical  Measurable 
Attributes 

3.2  Practical  Attribute 
Measures 

3.3  Performance 
Requirements,  Specific 

3.4  Performance 
Criteria,  Specific 

3.5  Measurement  Procedures 


2.1  The  system  under  discussion  in  this  manual 
is  tank  gunnery  and  includes  task 
descriptions  for  all  crew  members — drivers, 
loaders,  gunners,  and  tank  commanders. 

2.2  The  mission  of  this  system  was  to  score  a 
first-round  target  hit  in  the  minimum 
possible  time. 

3.1  Individual  tasks  for  crew  members  were 
presented.  These  tasks  apply  to  nearly  all 
series  and  models  of  tanks. 

3.2  Crew  standards  were  presented  for  each 
series  and  model  of  tank.  Performance 
standards  included:  time  to  fire,  range, 
degrees  off-target  for  daylight  hours  and 
darkness  under  artificial  illumination. 
Skill  tests  and  scoring  sheets  for 
performance  tests  are  included  in  this 
manual. 


4.  Planning  Components 
of  the  Process 

4.1  Analytic  Methods 

4.2  Parameter  Determinations 

4.3  Apparatus  for  Testing 

4.4  Personnel  for  Testing 

4.5  Test  Plans 


5.  Application  Components 
of  the  Process 

5.1  Test  Execution 

5.2  Data  Analysis 

5.3  Findings  Interpretation 

5.4  Conclusions  and 
Recommendations 


6.  Further  Research  Areas 

6.1  Measurement  System 
Limitations 

6.2  Research  Potentials/ 
Pricrities 

6.3  Research  Planning 
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1  Topics  Relevant  i  2 
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1.  St«te  of  the  Art  Review  2.1 

of  the  Process 

1.1  General  System 
Measurements 

1.2  System  Taxonomy 
Model  (STM) 

1.3  Overall  Conceptual 
Process  Model  (CPM) 

2.  Contextual  Components 
of  the  Process 

2.1  System  Definition 

2.2  Mission  Definition 

2.3  Environment  Definition 

2. H  General  Constraints 

2.5  Performance 
Requirements,  Ultimate 

2.6  Performance 
Criteria,  Ultimate 

3.  Analytic  Components 
of  the  Process 

3.1  Practical  Measurable 
Attributes 

3.2  Practical  Attribute 
Measures 

3.3  Performance 
Requirements,  Specific 

3. *l  Performance 

Criteria,  Specific 
3.5  Measurement  Procedures 

*1.  Planning  Components 
of  the  Process 
*1.1  Analytic  Methods 
i| .  2  Parameter  Determinations 
*1.3  Apparatus  for  Testing 
H.A  Personnel  for  Testing 
A. 5  Test  Plans 

5.  Application  Components 
of  the  Process 

5.1  Test  Execution 

5.2  Data  Analysis 

5.3  Findings  Interpretation  3.2 
5. *l  Conclusions  and 

Recommendations 

6.  Further  Research  Areas 

6.1  Measurement  System 
Limitations 

6.2  Research  Potentials/ 
Priorities 

6.3  Research  Planning 


The  various  factors  that  go  into  the 
optimization  of  tank  weaponry  effectiveness 
were  broken  down  into  subsystems  and 
analyzed  regarding  the  impact  of  each 
subsystem  -  and  subsystem  integration  -  on 
the  combat  capability  of  the  total  tank 
weapons  system.  Subsystems  were  defined 
into  personnel,  logistics,  training, 
development,  and  management  components  and 
are  examined  in  turn. 

3.1  The  characteristics  of  each  subsystem 
to  be  measured  were  as  follows: 

(1)  Personnel  Management  System, 
consisting  of  training,  distribution, 
sustainment,  separation,  and 
management  components. 

(2)  Logistics  System,  consisting  of 
fixing,  arming,  fueling,  and  managing. 

(3)  Training,  consisting  of  entry-level 
individual  training,  collective  and 
combined  arms  training,  reserve 
component,  training  management,  tank 
distribution/procurement  requirements , 
and  organization/doctrine. 

(4)  Tank  Forces  Management  System, 
including  science  and  technology  base, 
tank  system  development,  and  resource 
allocation. 

Units  of  measurement,  of  course,  differ 
between  subsystems.  The  personnel 
subsystem  consisted  of  processes  directed 
toward  the  procurement,  training, 
utilization,  separation,  development,  and 
motivation  of  military  personnel.  The 
training  requirements  of  the  personnel 
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management  system  are  separated  into  basic 
combat  training,  advanced  individual 
training,  and  one-station  unit  training. 

The  distribution  aspect  involving  the 
proper  mix  of  personnel  to  fill  field 
requirements  was  assessed  by: 

(1)  Properly  identified  and  timely 
submitted  requirements  by  units. 

(2)  Accurate  authorizations  and  personnel 
inventory  data  bases. 

(3)  Support  at  all  oommand  levels  through 
the  reduction  of  diversions. 

The  sustainment  component  of  the  Personnel 
Management  System  (to  d^v.lop  and  maintain 
the  career  force  to  be3t  meet  the  readiness 
requirement)  was  assessed  through  the 
Enlisted  Force  Management  Plan  (EFMP), 
which  provides  both  qualitative  and 
quantitative  goals  for  the  Army  for  the 
period  FY  73  -  FY  82. 

The  logistics  subsystem,  which  seeks  to 
optimize  the  interaction  of  functional 
equipment,  trained  personnel,  and 
responsive  support,  was  assessed  through 
the: 

(1)  Fixing  or  supply  and  maintenance 
system  (wholesale  and  retail  logistics 
doctrine  and  procedures). 

(2)  Arming  requirements  for  the  tank 
force,  e.g.,  ammunition  resupply 
rates,  transportation  requirements  for 
bulk  cargo  from  entry  points  to 
ultimate  user. 

(3)  Fueling  supply,  e.g.,  what 
organizations  at  what  level  fulfill 
this  function  and  what  report/request 
procedures  exist  to  support  it. 
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Assessment  of  the  training  subsystem  is 
made  through  evaluation  of  the  Army's 
service  schools  and  training  centers, 
training  materials  and  extension  courses, 
operational  unit-field  training,  and 
simulated  combat  training. 

Finally,  the  tank  forces  management  system 
is  based  on  an  analysis  of  system 
characteristics,  levels  of  decision, 
material  acquisition  analysis,  and 
operational  and  development  testing. 

The  findings  of  this  analytical  report  have 
direct  relevance  for  the  primary  mission  of 
the  Army  Tank  Development  Program: 
equipping  armor  units  with  the  best 
available  tank  in  sufficient  numbers  to 
counter  the  threat: 

(1)  There  currently  exists  no  viable  tank 
development/procurement  strategy  in 
the  Army. 

(2)  There  is  no  single  coordinating  agency 
within  the  U.S.  Array  Material 
Readiness  and  Development  Command 
(DARCOM)  in  charge  of  integrating  all 
tank  related  science  and 
technology-based  programs. 

(3)  Docunents  in  current  use  do  not 
identify  user  tank  development 
priorities . 

(4)  Control  of  tank  programs  i*i 
fragmented. 

(5)  Tank  project  managers  fail  to  initiate 
early  planning  in  the  development 
cycle. 

(6)  Responsible  Army  staff  agencies  are 
not  reviewing  logistic  plans  prior  to 
program  milestones. 


Van  Acker,  A.  A  Wohl,  J.G.  Modeling  the  sonar  operator's  detection  process: 
A  progress  report.  In  G.  W.  Levy  (Ed.),  Symposium  on  applied  model  of 
man-machine  systems  performance  (NR69H— 59 1 ) •  Columbus,  OH:  North  American 
Aviation,  November  1968.  (AD-697  939). 
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1.  State  of  the  Art  Review 
of  the  Process 

1 . 1  General  System 
Measurements 

1.2  System  Taxonomy 
Model  (STM) 

1.3  Overall  Conceptual 
Process  Model  (CPM) 

2.  Contextual  Components 
of  the  Process 

2.1  System  Definition 

2.2  Mission  Definition 

2.3  Environment  Definition 
2.*)  General  Constraints 

2.5  Performance 
Requirements,  Ultimate 

2.6  Performance 
Criteria,  Ultimate 


1.3  To  assist  in  developing  sonar  operator 

performance  standards,  models  representing 
different  aspects  of  an  operator's 
activities  are  being  developed  and 
implemented  as  computer  programs.  This 
paper  discusses  the  search  and  detection 
performance  model.  The  overall  simulation 
model  includes  three  submodels: 

(1)  The  geometric  submodel  (which  controls 
movement  of  the  target  submarine). 

(2)  The  physical  world  submodel 
(controlling  sonar  signals, 
reverberat-ons,  noise,  etc.). 


3.  Analytic  Components 
of  the  Process 

3.1  Practical  Measurable 
Attributes 

3.2  Practical  Attribute 
Measures 

3.3  Performance  2.M 

Requirements,  Specific 

3-1)  Performance 

Criteria,  Specific 
3.5  Measurement  Vrocedures 

H.  Planning  Components 
of  the  Process 
1).1  Analytic  Methods 
<1  2  Parameter  Determinations 
1).3  Apparatus  for  Testing 
Personnel  for  Testing 
i).5  Test  Plans 


(3)  The  decision-maker  submodel  (which 
simulates  the  operator's  search  and 
detection  processes  and  procedures). 

The  decision-maker  submodel  involves 

certain  basic  assumptions: 

(1)  The  operator  fixates  on  the  CRT  at  a 
series  of  points  in  consecutive 
sequence,  and  a  target  can  be  seen 
only  when  fixating. 

(2)  A  specified  scanning  policy  is 
followed  in  regard  to  fixation. 


Only  targets  (not  noise)  will  cause 
interruption  of  the  scan  policy. 

5.2  Data  Analys' s 

5.3  Findings  Interpretation 
5. *l  Conclusions  and 

Recommendations 


Application  Components 

of  the  Process 

5.1  Test  Execution 


(3) 


6.  Further  Research  Areas 

6.1  Measurement  System 
Limit**' ions 

6.2  Research  Potentials/ 
Priorities 

6.3  Research  Planning 
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(4)  Probability  of  glimpsing  a  target  is  a 
function  of  retinal  position  and 
brightness  contrast  ratio. 


(5)  Probability  of  glimpsing  a  target  also 
is  a  function  of  an  internal  criterion 
used  by  the  operator  (which  depends  on 
his  conservatism,  short-term  memory, 
data  handling  capacity,  the  noise 
level  on  the  PPI,  as  well  as  some 
psychophysical  limitations). 

(6)  The  decision  maker  decides  that  a 
target  is  present  if  his  last  glimpse 
of  the  target  is  foveal  and  if  he  has 
glimpsed  it  a  specified  number  of 
times  out  of  the  last  group  of  pings 
(called  the  detection  announcement 
factor) . 

4.2  As  variable  inputs,  the  decision-maker 
submodel  has  the  following: 

(1)  A  scanning  policy. 

(2)  Fixation  time. 

(3)  Glimpse  criterion  factor  (the 
operator's  internal  criterion). 

(4)  Detection  announcement  factor. 

5.3  Early  results  with  the  submodel  indicated 
that  fixation  time  is  not  a  highly  critical 
factor  to  detection  range,  and  that  various 
regular  search  patterns  produced  little 
statistical  differences.  However,  the 
sonar  performance  was  extremely  sensitive 
to  the  operator's  internal  glimpse 
criterion,  and  also  to  the  detection 
announcement  factor . 

6.1  The  basic  detection  model  involves  only  the 
video  sonar  scan  and  applies  only  to  a 
small  proportion  of  the  situations 
encountered  in  practice  (viz.,  isothermal 
water  and  non-maneuvering  targets).  Work 
is  underway  to  extend  the  model  and  remove 
these  constraints. 
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1 . 1  General  System 
Measurements 

1.2  System  Taxonomy 
Model  (STM) 

1.3  Overall  Conceptual 
Process  Model  (CPM) 

2.  Contextual  Components 

of  the  Process 

2.1  System  Definition 

2.2  Mission  Definition 

2.3  Environment  Definition  2.2 

2.4  General  Constraints 

2.5  Performance 
Requirements,  Ultimate 

2.6  Performance 
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Attributes 
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Requirements,  Specific 
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Utility  theory  is  a  division  of  measurement 
theory  that  assigns  numbers  to  objects  by 
indexing  them  to  a  decision-maker's  values. 
Utility  theory,  however,  unlike  measurement 
theory  in  general  -  differs  by  measuring 
objects  of  cost  or  value,  relates  objects 
by  means  of  preferences,  while  operations 
involving  these  objects  are  either  missing 
or  are  made  through  surrogate  objects. 

The  goal  of  this  analysis  of  utility  theory 
is  in  its  application  to  certain  decision 
situations  that  may  be  classified  according 
to  the  following  factors: 

(1)  Static  versus  dynamic  decision 
environment. 

(2)  Single  decision  maker  versus  multiple 
decision  makers. 

(3)  Single  aspect  choice  entity  versus 
multiple  aspect  choice  entity. 


4.  Planning  Components 
of  the  Process 

4.1  Analytic  Methods 

4.2  Parameter  Determinations 

4.3  Apparatus  for  Testing 

4.4  Personnel  for  Testing 

4.5  Test  Plans 


Specifically,  five  model  classes  of  utility 
theory  are  discussed:  "weak  orders", 
"difference  measurement",  "bisymmetric 
measurement",  "conjoint  measurement",  and 
"expected  utility  measurement". 


5.  Application  Components 
of  the  Process 

5.1  Test  Execution 

5.2  Data  Analysis 

5.3  Findings  Interpretation 

5.4  Conclusions  and 
Recommendations 


3.2  The  weak  order  measurement  model  is  applied 
in  cases  of  multiple  affected  individuals 
and  assumes  transitivity  of  preferences. 


6.  Further  Research  Areas 

6.1  Measurement  System 
Limitations 

6.2  Research  Potentials/ 
Priorities 

6.3  Research  Planning 
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Difference  models  compare  the  relative 
differences  of  the  strength  of  preference 
between  pairs  of  choice  entities,  and  uses 
"operational  surrogates". 

The  idea  behind  bisymmetric  measurement  is 
to  find  an  element  "c"  that  bisects  "a"  and 
"b",  i.e.,  the  objective  is  that  "c"  has 
the  average  utility  of  "a"  and  "b" 
numerically  speaking. 

Conjoint  measurement  models  are  used  for 
measuring  utilities  over  several  choice 
entities  that  take  on  different  value 
configurations. 

Expected  utility  theory  makes  three  common 
assumptions  about  preferences  among  risky 
choice  entities: 

(1)  Risky  alternatives  can  be  ordered 
transitively. 

(2)  If  events  have  common  outcomes,  then 
preferences  among  risky  alternatives 
should  be  independent  of  those  events. 

(3)  There  are  equivalents  for  all  possible 
uncertain  entities. 

5.4  Decision  analysts  may  improve  their 

model-building  by  using  utility  theory  to 
define  problem  structure,  model  forms  and 
possible  errors  in  models.  However, 
utility  theory  is  not  appropriate  for  scale 
construction. 
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1.3  Overall  Conceptual  P  P 

Process  Model  (CPM)  c'c 

2.  Contextual  Components 
of  the  Process 

2.1  System  Definition 

2.2  Mission  Definition 

2.3  Environment  Definition 

2. 4  General  Constraints 

2.5  Performance 
Requirements,  Ultimate 

2.6  Performance  2.4 

Criteria,  Ultimate 

3.  Analytic  Components 
of  the  Process 

3.1  Practical  Measurable 
Attributes 

3.2  Practical  Attribute 
Measures 

3.3  Performance  3«1 

Requirements,  Specific  __j 

3. A  Performance 

Criteria,  Specific  3«2 

3.5  Measurement  Procedures 


This  system  is  an  automated  flight  training 
system  consisting  of  an  instrument  flight 
maneuvers  flight  simulator  modified  to 
operate  with  three  measurement  systems. 

The  system  was  configured  as  a  high 
performance  fighter  aircraft  (F-4E)  capable 
of  four  instrument  flight  maneuvers.  The 
flight  conditions  selected  are  straight  and 
level  flight,  standard  rate  climbs  and 
descents,  level  turns  and  climbing  and 
descending  turns. 

The  system  consists  of  an  F-4E  cockpit 
mounted  on  a  four-degree  of  freedom  motion 
platform.  It  contains  all  controls  and 
displays  of  the  jet  fighter  cockpit  except 
radio  navigation,  communications  and 
weapons  control. 

The  following  characteristics  and  units  of 
measure  were  used  in  this  study: 


Planning  Components 

of  the  Process 

4.1  Analytic  Methods 


PARAMETER 


UNITS 


4.2 

Parameter  Determinations 

4.3 

Apparatus  for  Testing 

(1) 

System  Clock  Count 

4.4 

Personnel  for  Testing 

4.5 

Test  Plans 

(2) 

Elevator  Stick  Force 

Pounds 

Application  Components 
of  the  Process 

5.1  Test  Execution 

(3) 

Elevator  Stick  Dis¬ 

Inches 

5.2 

Data  Analysis 

placement 

5.3 

Findings  Interpretation 

5.4 

Conclusions  and 
Recommendations 

(4) 

Angle  of  Attack 

Units 

6.  Further  Research  Areas 

6.1  Measurement  System 
Limitations 

6.2  Research  Potentials/ 
Priorities 

6.3  Research  Planning 
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(5) 

Pitch  Attitude 

Degrees 

(6) 

Climb/Descent  Rate 

Feet  per  Min 

(7) 

Altitude 

Feet 

(8) 

Right  Throttle  Dis¬ 
placement 

Degrees 

(9) 

Airspeed 

Knots 

(10) 

Aileron  Stick  Force 

Pounds 

(11) 

Aileron  Stick  Dis¬ 
placement 

Inches 

(12) 

Roll  Attitude 

Degrees 

(13) 

Turn  Rate 

Degrees  per 
Second 

(14) 

Heading 

Degrees 

(15) 

Rudder  Pedal  Force 

Pounds 

(16) 

Rudder  Pedal  Dis¬ 
placement 

Inches 

(17) 

Sideslip 

Degrees 

(18) 

Turbulent  Air 
Intensity 

Arbitrary  Units 

Initially,  16  of  these  measures  were 
selected  for  use.  Correlation  analysis  of 
redundant  information  reduced  the  16  to  12. 
Multiple  discriminant  analysis  produced 
nine  measures  which  could  be  weighted  and 
sunned  into  a  single  score. 


All  data  were  recorded  on  magnetic  tape  at 
the  rate  of  five  times-per-second  in  real 
time  for  each  full  training  run. 
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4.1  Correlation  and  multiple  discriminant 
analyses  were  used  on  the  collected 
measurement  data. 

4.2  The  controlled  test  parameters  were  air 
turbulence  set  at  "light"  for  this  test, 
aircraft  weight  was  either  light  or  heavy 
and  a  center-of-gravity  shift  of  29.0  to 

30.2  percent  mean  aerodynamic  chord. 

4.3  The  test  equipment  was  the  Training  Device 
Computer  System  (TRADEC)  located  at  the 
Naval  Training  Equipment  Center,  Orlando, 
Florida.  It  was  configured  as  a  fixed  wing 
aircraft  (F-4E)  on  a  four-degree  of  freedom 
motion  platform. 

4.4  The  test  subjects  were  twelve  low-time 
civilian  student  and  private  pilots. 

5.4  The  major  conclusion  and  recommendation  of 
the  study  was  that  the  discriminant  model 
should  be  applied  to  the  problem  of 
specifying  measures  for  future  flight 
training  systems.  In  addition,  a  series  of 
individual  conclusions  and  recommendations 
were  presented. 
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1.  State  of  the  Art  Review 

1.3 

The  implementation  of  the  measurement 

of  the  Process  system  requires: 

1.1  General  System 
Measurements 


1.2  System  Taxonomy 
Model  (STM) 

1.3  Overall  Conceptual 
Process  Model  (CPM) 

2.  Contextual  Components 
of  the  Process 

2.1  System  Definition 

2.2  Mission  Definition 

2.3  Environment  Definition 

2.4  General  Constraints 

2.5  Performance 
Requirements,  Ultimate 

2.6  Performance 
Criteria,  Ultimate 


(1)  Definition  of  criterion  objectiven  in 
terms  of  a  candidate  set  of  simulated 
parameters. 

(2)  Evaluation  of  the  proposed  set  of 
measures  for  the  purpose  of  validation 
and  simplification. 

(3)  Specification  of  criterion  performance 
by  requiring  experienced  instructor 
pilots  to  fly  the  particular  maneuver. 


3.  Analytic  Components 
of  the  Process 

3.1  Practical  Measurable 
Attributes 

3.2  Practical  Attribute 

Measures  P  . 

3.3  Performance  • 

Requirements,  Specific 

3.4  Performance 
Criteria,  Specific 

3.5  Measurement  Procedures 

4.  Planning  Components 
of  the  Process 

4.1  Analytic  Methods  ^ 

4.2  Parameter  Determinations 

4.3  Apparatus  for  Testing 

4.4  Personnel  for  Testing 

4.5  Test  Plans 

5.  Application  Components 
of  the  Process 

5.1  Test  Execution 

5.2  Data  Analysis 

5.3  Findings  Interpretation 

5.4  Conclusions  and 
Recommendations 


(4)  Collection  of  normative  data  using 

students  as  they  progress  through  the 
training  program. 

The  Advanced  Simulation  in  Undergraduate 
Pilot  Training  (ASUPT)  facility  is  designed 
to  be  a  research  device  capable  of 
providing  answers  regarding  the  hardware 
design  and  effective  use  of  flight 
simulators . 

A  criterion  referenced  approach  to 
measurement  system  development  was  pursued 
under  the  following  constraints: 

(1)  Measures  will  assess  the  degree  to 
which  the  criterion  objectives  are 
met. 


6.  Further  Research  Areas 

6.1  Measurement  System 
Limitations 

6.2  Research  Potentials/ 
Priorities 

6.3  Research  Planning 
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(2)  Measures  will  reflect  only  the  most 
salient  characteristics  of 
performance. 

(3)  Measures  will  be  meaningful  and 
interpretable  to  the  user  —  the 
student  and  instructor  pilot. 

(4)  Measures  will  be  generated  on  a 
real-time  basis  so  that  feedback  is 
immediate. 


3.1  Those  attributes  measured  in  this  study 
were: 


Altitude 

Airspeed 

Heading 

Stick  movement 
Throttle  movement 
Elevator  stick  force 


Pitch  rate 
Pitch  acceleration 
Roll  rate 
Roll  acceleration 
Vertical  velocity 
Vertical  accelera¬ 
tion 


4.1  The  statistics  in  this  study  were 

calculations  of  the  mean,  mean  root  square, 
and  the  standard  deviation. 
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The  purpose  of  the  Weapon  System 
Effectiveness  Industry  Advisory  Committee 
was  to  provide  technical  guidance  and 
assistance  to  Air  Force  System  Command  in 
the  development  of  a  technique  to  apprise 
management  of  current  and  predicted  weapon 
system  effectiveness  at  all  phases  of 
weapon  system  life. 

It  was  noted  that  system  effectiveness 
evaluation/prediction  can  be  reduced  to  an 
ordered  set  of  tasks  as  follows: 

(1)  Mission  Definition 

a.  Functional  description  of 
purpose 

b.  Quantitative  requirements 

(2)  System  Description 

a.  General  configuration 

b.  Block  diagram 

c.  Time  line  analysis 

(3)  Specification  of  Figures  of  Merit 

(4)  Identification  of  Accountable  Factors 

a.  Level  of  accountability 

b.  Personnel  characteristics 

c.  Procedure  characteristics 
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d.  Hardware  characteristics 


e.  Logistics 


f.  Data  constraints 


(5)  Model  Construction 

a.  Assumptions 

b.  Delineation  of  possible  mission 
outcome 

c.  Delineation  of  significant  system 
states 

(6)  Data  Acquisition 

a.  Specification  of  data  elements 

b.  Specification  of  test  methodology 

c.  Specification  of  data  reporting 
system 

(7)  Parameter  Estimation 

(8)  Model  Exercise 

a.  Numerical  evaluation  of 
effectiveness  and  its  factors 

b.  Comparative  system  analysis 

c.  Parameter  variation  study 

The  examples  of  Volumes  II  and  III  adhere 
rather  closely  to  this  analysis  of  the 
steps  required  to  achieve  a  system 
sffectiveness  evaluation/prediction. 

2.1  The  life  cycle  of  a  system  was  divided  into 
the  conceptual,  definition,  acquisition, 
and  operational  phases. 

3.1  System  effectiveness  was  defined  as  the 
function  of  a  system's  availability, 
dependability,  and  capability. 
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Availability  was  a  measure  of  the  system 
condition  at  the  start  of  the  mission.  It 
was  a  function  of  the  relationships  among 
hardware,  personnel  and  procedures. 

Dependability  was  a  measure  of  the  system 
condition  at  one  or  more  points  during  the 
mission  given  the  system  condition(s)  at 
the  start  of  said  mission. 

Capability  was  a  measure  of  the  ability  of 
the  system  to  achieve  the  mission 
objectives,  given  the  system  condition(s) 
during  the  mission.  Capability  accounts 
for  the  performance  spectrum  of  a  system. 

4.1  No  specific  statistics  were  given. 

However,  the  procedure  went  as  follows: 
effectiveness  was  the  product  of 
availability,  dependability  and  capability. 
To  provide  for  the  treament  of  the  various 
conditions,  these  three  variables  were 
expressed  as  a  vector  or  matrix.  In  highly 
complex  systems,  when  matrices  were 
impractical,  an  analog  or  digital  computer 
simulation  was  employed. 
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This  document  is  part  of  the  final  report 
of  the  Weapon  System  Effectiveness  Industry 
Advisory  Committee  (WSEIAC).  The  purpose 
of  the  Weapon  System  Effectiveness  Industry 
Advisory  Committee  was  to  provide  technical 
guidance  and  assistance  to  air  force  system 
command  in  the  development  of  a  technique 
to  apprise  management  of  current  predicted 
weapon  system  effectiveness  at  all  phases 
of  weapon  system  life.  Four  systems  were 
illustrated  including  an  airborne  avionics 
system,  an  intercontinental  ballistic 
missile  system,  a  long  range  radar 
surveillance  system,  and  a  spacecraft 
system.  The  four  phases  of  system  life 
(conceptual,  definition*  acquisition, 
operation)  were  discussed  as  well  as  the 
eight  tasks  used  to  evaluate  system 
effectiveness. 


(1) 

Mission  definition 

(2) 

System  description 

(3) 

Specification  of  figure(s)  of 

merit 

(4) 

Identification  of  accountable 

facto 

(5) 

Model  construction 

(6) 

Data  acquisition 

(7) 

Parameter  estimation 

(8) 

Model  exercise 

6.  Further  Research  Areas 

6.1  Measurement  System 
Limitations 

6.2  Research  Potentials/ 
Priorities 

6.3  Research  Planning 


A-471 


i  Topics  Relevant 

1  1 

1  1 

I  to  System  Development 

!  Topic! 

land  Evaluation  Technology 

i  No.  ! 

ABSTRACT 

2.1  The  long  range  radar  surveillance  system 
consisted  of  two  transmitters  in  parallel, 
an  antenna,  a  receiver,  a  display  and 
synchronizer,  and  an  operator. 

3.3  The  system  should  detect  target  aircraft 
above  the  horizon  line  of  sight  at  ranges 
up  to  200  miles  and,  while  the  target  is 
within  this  maximum  range,  track  it  in 
range  and  azimuth  within  admissible  error. 
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The  purpose  of  the  Weapon  System  Effective' 
ness  Industry  Advisory  Committee  was  to 
provide  technical  guidance  and  assistance 
to  air  force  system  command  in  the 
development  of  a  technique  to  apprise 
management  of  current  and  predicted  weapon 
system  effectiveness  at  all  phases  of 
weapon  system  life.  The  objective  of  Task 
Group  II  was  to  review  existing  documents 
and  recommend  uniform  methods  and 
procedures  to  be  applied  in  predicting  and 
measuring  systems  effectiveness  during  all 
phases  of  a  weapon  system  program. 


2.1  This  report  is  concerned  primarily  with 
four  examples  of  effectiveness  evaluation, 
involving  the  following  systems: 


(1) 

Avionics  system  in  a  tactical  fighter 
-  bomber. 

(2) 

Squadron  of  ICBMs. 

(3) 

Fixed  radar  surveillance 
evaluation  system. 

and  threat 

(4) 

A  spacecraft  system. 

Each  system  is  evaluated  at  a 
phase  of  development. 

different 

5.  Application  Components 
of  the  Process 

5.1  Test  Execution 

5.2  Data  Analysis 

5.3  Findings  Interpretation 
5. A  Conclusions  and 

Recommendations 

6,  Further  Research  Areas 

6.1  Measurement  System 
Limitations 

6.2  Research  Potentlala/ 
Priorities 

6.3  Research  Planning 
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1 .  State  of  the  Art  Review 
of  the  Process 

1.1  General  System 

2.1 

This  study  addressed  competing 
communications  systems. 

Measurements 

1.2  System  Taxonomy 

Model  (STM) 

1.3  Overall  Conceptual 

2.2 

The  purpose  of  these  systems  is  to  transfer 
information  between  two  separate  locations. 

Process  Model  (CPM) 

2.  Contextual  Components 
of  the  Process 

2.1  System  Definition 

2.2  Mission  Definition 

2.5 

The  requirement  is  that  the  information  is 
received  and  that  the  information  reaches 
its  destination. 

2.3  Environment  Definition 

2.4  General  Constraints 

2.5  Performance 
Requirements,  Ultimate 

2.6  Performance 

Criteria,  Ultimate 

2.6 

The  ultimate  criteria  are  that  the 
information  is  understood  within 
"acceptable  boundaries  of  quality  or  error 
rate"  and  that  it  reaches  its  destination 
in  a  timely  fashion. 

3.  Analytic  Components 
of  the  Process 

3.1  Practical  Measurable 
Attributes 

3.1 

The  measure  of  benefit  has  two  dimensions 
—  information  and  time. 

3.2  Practical  Attribute 
Measures 

3.3  Performance 

3.2 

Information  flow  and  information  capacity 
can  be  measured  mathematically. 

Requirements,  Specific 
3-4  Performance 

Criteria,  Specific 
3.5  Measurement  Procedures 

<1.  Planning  Components 
of  the  Process 


3.3  Specific  performance  requirements  of  this 
system  include:  transportability, 
mobility,  capacity,  quality  of  service, 
survivability,  vulnerability. 


4.1  Analytic  Methods 

4.2  Parameter  Determinations  4,1 

4.3  Apparatus  for  Testing 

4.4  Personnel  for  Testing 

4.5  Test  Plans 

5.  Application  Components 
of  the  Process 

5.1  Test  Execution 

5.2  Data  Analysis 

5.3  Findings  Interpretation 

5.4  Conclusions  and 
Recommendations 


An  integrated  system  effectiveness  model 
was  developed  in  this  study  which  is 
capable  of  providing  a  single  explicit 
measure  of  system  effectiveness  when 
evaluating  competing  communication  systems. 
This  model  is  based  on  statistical  analysis 
and  the  probability  of  successful 
communications  derived  from  queuing 
relationships. 


6.  Further  Research  Areas 

6.1  Measurement  System 
Limitations 

6.2  Research  Potentials/ 
Priorities 

6.3  Research  Planning 
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The  approach  was  broken  down  into  four 
tasks: 

(1)  Development  of  a  comprehensive  listing 
of  criteria  and  associated  performance 
factors. 

(2)  Establishment  of  the  format  for  three 
matrices:  (a)  assignment  of  mission  to 
conflict  intensity  by  frequency  of 
occurrence  and  relating  allocation  of 
resources  to  conflict  intensity;  (b) 
identification  of  tactical  functions; 
and  (c)  relating  quantitative 
communications  requirements  by 
communication  nets  to  tactical 
functions. 


(3)  Development  of  quantitative  relation¬ 
ships  between  the  criteria  and 
measures  of  effectiveness. 


(4)  Use  of  the  above  items  to  show  the 

procedure  for  measuring  effectiveness 
of  the  proposed  system. 

The  conclusion  of  this  report  states  that 
it  is  desirable  at  this  time  to  test  the 
model  and  evaluation  concept  by  means  of  a 
not  too  complex  test  problem.  It  is 
believed  that  a  test  problem  will  establish 
confidence  in  the  model  and  provide 
insights  into  the  model's  operation. 
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1.  State  of  the  Art  Review  1.1 

of  the  Process 

1.1  General  System 
Measurements 

1.2  System  Taxonomy 
Model  (STM) 

1.3  Overall  Conceptual 
Process  Model  (CPM) 

2.  Contextual  Components 
of  the  Process 

2.1  System  Definition 

2.2  Mission  Definition 

2.3  Environment  Definition 
2.1)  General  Constraints 

2.5  Performance 
Requirements,  Ultimate 

2.6  Performance 
Criteria,  Ultimate 

3.  Analytic  Components 
of  the  Process 

3.1  Practical  Measurable 
Attributes 

3.2  Practical  Attribute 
Measures 

3.3  Performance 
Requirements,  Specific 

3.4  Performance 
Criteria,  Specific 

3.5  Measurement  Procedures 

4.  Planning  Components 
of  the  Process 

4.1  Analytic  Methods 

4.2  Parameter  Determinations 

4.3  Apparatus  for  Testing 

4.4  Personnel  for  Testing 

4.5  Test  Plans 

5.  Application  Components 
of  the  Process 

5.1  Test  Execution 

5.2  Data  Analysis 

5.3  Findings  Interpretation 
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General  methodology  is  introduced  which, 
when  used  appropriately  by  decision  makers, 
may  assist  them  in  deciding  which  system 
concepts  among  several  to  approve  for 
further  development.  This  technical  note 
attempts  to  establish  a  general  framework 
around  which  manpower  factors  can  be 
effectively  introduced  into  system  analyses 
studies . 

It  is  stated  that  with  expert  inputs  from 
specialists  in  training,  selection  and 
utilization  of  personnel,  and  human 
performance,  a  human  factors  team  should  be 
able  to  provide  an  estimate  of  which 
concept  from  the  human  factors  viewpoint 
will  best  fulfill  system  performance 
requirements  at  least  total  cost. 

It  is  suggested  that  to  successfully 
complete  the  manpower  factors  part  of  the 
systems  analysis,  a  constantly  updated  data 
bank  of  manpower  resources  should  be 
available.  The  data  bank  would  include 
training  time  and  cost  data  on  all  past 
fielded  systems,  and  a  skill-level  breakout 
of  the  present  Army  performance  data  in  the 
areas  of  vision,  tasks,  etc. 

Experts  in  the  area  of  training  and  job 
analysis  should,  with  this  information,  be 
able  to  estimate  the  skills  required  for 
each  concept  being  considered.  If  the 
estimates  are  detailed  enough,  sufficient 
sensitivity  should  exist  to  differentiate 
among  the  concepts  being  considered. 


6.  Further  Research  Areas 
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In  addition,  appropriate  data  in  the  data 
bank  would  enable  estimates  to  be  performed 
as  to  whether  a  person  can  or  cannot 
perform  the  functions  required  in  each 
concept . 

Following  the  initial  system-concept 
comparison,  trade-off  studies  could  be 
conducted.  The  human  factors  team  could 
assist  in  deriving  these  data  in  various 
forms:  cost,  estimated  amount  of  down  time, 
system  performance  change  and  personnel 
skill-requirements  change. 
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Model  (STM) 

1.3  Overall  Conceptual 
Process  Model  (CPM) 


2.1  The  system  addressed  in  this  thesis  is  the 
Standard  Base  Supply  System  and  its 
associated  personnel. 

2.2  The  Base  Supply  System  provides  support  to 
aircraft  maintenance. 


2.  Contextual  Components  2.5 

of  the  Process 

2.1  System  Definition 

2.2  Mission  Definition 

2.3  Environment  Definition 
2. A  Ceneral  Constraints 

2.5  Performance 
Requirements,  Ultimate 

2.6  Performance  2.t 

Criteria,  Ultimate 

3.  Analytic  Components 
of  the  Process 

3.1  Practical  Measurable  3«1 

Attributes 

3.2  Practical  Attribute 
Measures 

3.3  Performance 
Requirements,  Specific 

3-A  Performance 

Criteria,  Specific 
3.5  Measurement  Procedures 

A.  Planning  Components 
of  the  Process 
A.1  Analytic  Methods 
A. 2  Parameter  Determinations 
A. 3  Apparatus  for  Testing  ^5 
A. A  Personnel  for  Testing 
A. 5  Test  Plans 

5.  Application  Components 
of  the  Process 

5.1  Test  Execution 

5.2  Data  Analysis 

5.3  Findings  Interpretation 
5. A  Conclusions  and 

Recommendations 


The  ultimate  performance  requirement  of 
this  system  is  that  the  base  supply  be 
controlled  in  the  most  cost-effective 
manner  to  ensure  that  resources  are  used 
efficiently  and  effectively. 

The  current  system  has  no  set  of  standard 
measurement  areas  to  indicate  how  well  the 
system  is  supporting  its  objectives. 

In  order  to  determine  which  attributes 
should  be  measured,  an  open-ended 
questionnaire,  accompanied  by  a  letter  of 
explanation  of  the  research  project,  was 
sent  to  the  selected  "experts."  The 
response  from  this  questionnaire  resulted 
in  the  development  of  a  "follow-on" 
questionnaire.  From  this  information  a 
consensus  of  opinion  on  the  measurable 
attributes  of  importance  was  obtained. 

The  measurement  tool  utilized  in  this  study 
was  the  Delphi  technique.  To  determine  a 
consensus,  the  mode  response  was  chosen  as 
it  is  the  response  which  occurs  most  often. 
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4.1  The  Delphi  technique  was  utilized  to  induce 
opinion  convergence  through  a  sequence  of 
questionnaires  using  controlled  feedback  to 
the  participants. 

"3 

4.4  It  was  determined  that  an  appropriate 
sample  size  for  the  study  would  be  35.  Two 
groups  were  selected  composed  of  supply 
experts  and  experts  from  the  maintenance 
career  field.  The  following  criteria  were 
used  as  a  basis  for  selection: 

a.  Stature  of  the  individual  in  his 
career  area. 

b.  Breadth  of  his  experience. 

c.  Degree  of  varied  assignments  with 
different  major  commands. 

d.  Level  of  assignment  from  base,  major 
commands  and  Air  Force  levels. 

5.4  A  consensus  was  obtained  on  seventeen 
measured  areas  from  the  panel  of  experts. 
This  initial  research  indicates  that  the 
Delphi  technique  can  be  effectively  used 
for  design  of  some  elements  of  a  management 
control  system,  but  there  are  limitations 
which  should  be  considered. 
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1.3  Overall  Conceptual 
Process  Model  (CPM) 

2.  Contextual  Components 
of  the  Process 

2.1  System  Definition 
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2.3  Environment  Definition 
2.0  General  Constraints 

2.5  Performance 
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Attributes 
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3. A  Performance 

Criteria,  Specific 
3.5  Measurement  Procedures 


1.1  This  paper  is  concerned  with  the 

development  of  models  which  will  aid  in  the 
design,  development,  test  and  evaluation  of 
military  systems,  equipment,  and 
facilities.  Too  often,  design  approaches 
that  are  taken  seem  to  be  based  on 
"clinical"  or  intuitive  Judgments  without 
formally  stated  assumptions.  Whenever 
assumptions  are  not  formally  stated  there 
is  an  increased  chance  that  erroneous 
assumptions  will  not  be  recognized  as  such. 
There  is  a  need,  as  model  builders,  to 
formalize  assumptions  to  be  sure  that  they 
are  not  inconsistent. 

1.3  This  paper  examines  the  feasibility  of  far 
more  sophisticated  digital  computer 
programs  [models]  than  currently  exist.  It 
is  possible  to  distinguish  several  classes 
of  models  which  would  be  extremely  useful 
if  they  (a)  existed  and  (b)  worked.  These 
include: 

(1)  Instrument  design/evaluation  programs. 
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A. 5  Test  Plans 


(2)  Human  operator  simulator  programs. 

(3)  Optimization  programs. 

(H)  Training  requirements  programs. 


.5.  Application  components  3.1  Major  subsections  of  human  operator 

of  the  Process  simulators  are  identified: 

5.1  Test  Execution 

5.2  Data  Analysis 

5.3  Findings  Interpretation 
5. A  Conclusions  and 

Recoenendatlons 
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(1)  Individual  Differences—The  model 

should  have  the  option  of  reading  in  a 
profile  of  abilities  so  that  a  system 
could  be  tested  ,o  determine  its 
sensitivity  to  various  operator 
states. 


(2)  Procedures - Procedures  for  the 

simulated  operator  must  be  oriented  to 
the  "perceptual"  level  and  stated  in 
terms  of  desires  rather  than 
imperatives.  The  model  can  never 
positively  require  tor  guarantee]  that 
the  operator  will  absorb  any 
particular  information  or  make  any 
particular  control  manipulation.  The 
locus  for  decisions  to  actions  resides 
in  the  operator,  not  in  external 
conmands  or  procedures. 

H.3  The  use  of  a  Human  Operator  Procedures 
(HOPROC)  language  for  Human  Operator 
Simulator  Programs  is  discussed. 
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of  the  Process 

4.1  Analytic  Methods 
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4.3  Apparatus  for  Testing 

4.4  Personnel  for  Testing 

4.5  Test  Plans 

5.  Application  Components 
of  the  Process 

5.1  Test  Execution 
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5.4  Conclusions  and 
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6.  Further  Research  Areas 
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This  thesis  addresses  a  method  which 
provides  a  basis  for  the  selection  of 
critical  attributes  which  best  discriminate 
between  acceptable  and  unacceptable  systems 
in  order  to  facilitate  the  selection  of 
measures  of  effectiveness. 

The  author  states  that  current  test 
structure  in  operational  testing  is  not 
amenable  to  the  standard  application  of 
multivariate  statistics.  The  methodology 
developed  in  this  thesis  encompasses  a 
means  to  combine  results  from  past  tests 
with  subjective  information  to  determine, 
the  relationship,  in  terms  of  covariances 
between  two  attributes.  This  information 
is  incorporated  with  subjectively  obtained 
acceptable  and  unacceptable  mean  vectors  in 
stepwise  discriminant  analysis. 

It  is  concluded  that  multivariate 
techniques  may  be  a  valuable  aid  in 
determining  which  attributes  contribute 
more  in  distinguishing  between  successful 
and  unsuccessful  systems.  It  is  also 
concluded  that  current  test  design  for 
operational  testing  can  be  modified  to 
facilitate  a  broader  use  of  multivariate 
statistical  analysis  techniques.  This 
modification  should  permit  the  correlation 
among  attributes  to  be  objectively 
determined  and  the  marginal  normality  of 
observations  for  each  attribute  to  be 
validated. 
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Tasks  performed  by  human  operators, 
technicians,  and  ground  crews  in  assembly, 
test,  and  handling  frequently  have  a 
significant  e/fect  on  the  efficiency  of  a 
weapon  system.  In  assessing  system  and 
design  feasibility,  one  must  also  determine 
the  reliability  of  human  performance. 

Methods  developed  for  estimating  human 
performance  reliability  require  that  the 
human  task  be  divided  into  discrete  steps, 
and  that  a  probability  model  be  fitted  to 
the  task.  Probability  values  of  successful 
performance  are  estimated  for  each  step  in 
the  model,  and  the  success  probability  for 
the  total  task  is  then  computed. 

If  the  task's  discrete  steps  are 
independent,  human  performance  reliability 
can  be  estimated  without  undue  difficulty. 
Data  stores  exist  which  provide  marginal 
probabilities  for  independent  task  steps 
(e.g.,  American  Institute  for  Research  data 
store) .  One  finds  that  the  great  majority 
of  operational  procedures  tasks  encountered 
break  down  into  dependent  steps.  The 
combination  of  dependent  relationships 
usually  is  unique  and  the  analyst  finds 
that  neither  data  nor  procedures  for 
estimating  the  dependent  probabilities  are 
available  to  him.  One  must  conclude  that 
the  conditional  probabilities  of  a  model 
composed  of  dependent  events  will  not  be 
found  in  a  data  store. 
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6.2  The  problem  of  estimating  conditional 

probabilities  of  dependent  task  steps  can 
only  be  solved  by  developing  transition 
models  that  make  the  transition  from 
the  marginal  probabilities  of  the  data 
store  to  the  conditional  probabilities  of 
the  dependent  relationships.  Although  much 
work  remains  to  be  done,  one  can  determine 
the  general  form  of  the  transition  models 
by  using  the  techniques  of  experimental 
design  and  analysis.  Two  major  problems 
that  must  be  solved  before  there  will  be 
significant  progress  in  developing 
transition  models  are  (1)  factors 
responsible  for  dependent  relationships 
among  task  steps  must  be  fully  identified, 
and  (2)  effects  of  dependent  relationships 
must  be  determined. 
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This  report  develops  a  practical  munitions 
effectiveness  methodology  relevant  to  five 
types  of  munitions:  artillery,  tank 
ammunition,  rockets,  mortars  and  mines. 

Since  no  formalized  procedure  exists  for 
making  comparative  munition  effectiveness 
analyses,  the  authors  here  attempt  to 
standardize  the  effectiveness  evaluation 
procedure. 

The  ultimate  goal  here  is  development  of  a 
standardized  formula  applicable  to  muni¬ 
tions  effectiveness  analyses  in  general, 
based  on  three  general  components:  defini¬ 
tion  of  weapon  purpose,  identification  and 
measurement  of  performance  and  "pain" 

(costs  of  achieving  a  given  level  of  per¬ 
formance),  and  determination  of  the  ex¬ 
change  relationships  among  quantities 
represented  by  the  above  concepts. 

The  effectiveness  methodology  developed  is 
applicable  to  the  five  classes  of  weapons 
mentioned  above,  applies  in  all  types  of 
weather  and  in  all  terrains. 

The  ultimate  requirement  of  this  munitions 
effectiveness  model  is  the  determination  of 
the  expected  fraction  of  the  target 
destroyed,  which  is  the  formal  definition 
of  munitions  effectiveness  (E). 


6.  Further  Research  Areas 

6.1  Measurement  System 
Limitations 

6.2  Research  Potentials/ 
Priorities 

6.3  Research  Planning 
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2.6  For  a  single  round,  munitions  effectiveness 

is  defined  by  the  equation  E  =  ARE(c),  where 

A  =  Availability 

R  =  Reliability  and 

E(c)  =  Effective  coverage  AB158 

3.1  Attributes  of  the  munitions  effectiveness 

and  concept  and  their  operationalization,  where 

3.2  quantifiable,  are  as  follows: 

(1)  Safety:  concerned  with  personnel 
safety  and  the  possibility  of 
premature  detonations. 

(2)  Reliability:  assessment  of  constitu¬ 
ent  elements  of  the  munition  in  terms 
of  performance,  both  human  and  weapon; 
also  includes  concepts  of  success  and 
failure. 

(3)  Availability:  includes  logistic 
integrity  of  personnel  -  weapons 
system;  manufacture,  availability  of 
raw  materials,  availability  of  plant 
capacity,  shipping,  storage,  and 
transport  to  the  unit  of  utilization, 
and  ability  of  weapon  to  survive  troop 
handling;  also  includes  concepts  of 
dependability,  durability,  ease  of  use 
and  training. 

(4)  Lethality:  capability,  arming  and 
delivery  accuracy,  kill  mechanisms  of 
overpressure,  shock,  penetration  or 
fragmentation,  etc. 

(5)  Range:  treated  by  assessing  overall 
effectiveness  as  a  summation  of 
effectiveness  at  specific  targets  at 
different  ranges  weighted  by  the 
probability  of  frequency  of 
occurrence. 

(6)  Flexibility:  the  degree  to  which  a 
design  can  be  modified  for  utilization 
in  different  procedures. 
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(7)  Countermeasures:  enemy  tactics  which 
affect  target  definition.  Examples 
include  armor  plating,  reenforced 
bunkers,  etc. 

(8)  Utilization  Rate:  rate  of  fire. 

(9)  Weight:  of  weapon. 

(10)  Firing  Doctrine  and  Tactics: 
effectiveness  of  individual 
subcomponents  of  man-weapon  system. 

(11)  Complexity:  simplicity  and  ease  of 
use  are  adversely  affected  by 
complexity. 

(12)  Training  and  Documentation:  the 
likelihood  of  successful  performance 
of  necessary  tasks,  degree  of 
training,  etc.,  i.e.,  the  human 
element. 

(13)  Cost:  referred  to  as  the  "pain.” 

A  sample  effectiveness  evaluation  for  the 

Lightweight  Company  Mortar  System  is  given: 

the  specific  steps  involved  in  the 

evaluation  are  as  follows: 


(1)  Step  1:  Define  Munition  Purpose. 

158 

(2)  Step  2:  Define  the  Mission:  general 
description,  functional  examination 
(success  criteria)  for  each 
component/module/assembly. 

(3)  Step  3:  Create  the  Availability/Relia 
bility  Event  Sequence  and  General 
Organizational  chart. 

(4)  Step  4:  Determine  Numerical  Estimates 
for  Events/Components  (Reliability 
Allocation  and  Prediction). 
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(5)  Step  5:  Calculate  Availability/Relia¬ 
bility  Estimates. 

(6)  Step  6:  Obtain  Target  and  Lethality 
Data. 

(7)  Step  7:  Exercise  Effectiveness 
Computer  Model. 
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The  general  goal  of  this  study  is  to 
provide  a  methodology  which  will  enable 
cognizant  persons  to  obtain  quantitative 
information  on  personnel  effectiveness  and 
relative  costs. 

The  system  under  discussion  in  this  report 
is  the  human  component  of  a  system  who 
performs  operator  or  maintenance  functions. 

The  mission  of  the  humen  component  in  this 
system  is  that  his/her  function  is 
performed  adequately  and  in  such  a  way  that 
it  will  always  lead  towards  mission 
accomplishment. 


3.  Analytic  Components 
of  the  Process 

3.1  Practical  Measurable 
Attributes 

3.2  Practical  Attribute 
Measures 

3 . 3  Performance 
Requirements,  Specific 

3.4  Performance 
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3-5  Measurement  Procedures 


As  there  exists  no  body  of  quantitative 
evidence  about  the  performance  effective¬ 
ness  of  personnel  in  present  systems,  it  is 
suggested  that  as  a  first  step  it  would  be 
appropriate  to  collect  data  to  be  used  as  a 
basis  for  predictions.  Samples  should  be 
selected  which  will  generalize  to  entire 
classes  of  populations. 


4.  Planning  Components 
of  the  Process 

4.1  Analytic  Methods 

4.2  Parameter  Determinations 

4.3  Apparatus  for  Testing 

4.4  Personnel  for  Testing 

4.5  Test  Plans 

-5.  Application  Components 
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5.1  Test  Execution 
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It  is  suggested  that  the  following 
parameters  should  be  used  to  describe  the 
operating  conditions  for  which  task 
accomplishment  must  be  predicted: 

(1)  Number 

(2)  Sequence 

(3)  Response  pacing  (operator  pacing) 


6.  Further  Research  Areas 

6.1  Measurement  System 
Limitations 

6.2  Research  Potentials/ 
Priorities 

6.3  Research  Planning 
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(4)  Feedback 

(5)  Required  Accuracy 

(6)  Display  exposure  time 

(7)  Display  visibility 

(8)  Type  of  stimulus  in  a  display 

(9)  The  number  of  stimulus  dimensions 

(10)  The  number  of  visual  stimuli  displayed 

(11)  Operator  function 

(12)  Stimulus  movement 

(13)  Requirements  for  control  -  display 
coordination. 

The  plan  suggested  for  research  is: 

(1)  Select  parameters  and  start 
observation  on  simulated  system. 

a.  Contact  system  designers  regard¬ 
ing  useful  content  and  format  of 
data  and  reported  PPE  indices. 

b.  Utilize  the  automated  OSD. 

(2)  Test  and  refine  parameters. 

(3)  Contacts  to  determine  how  methodology 
might  be  implemented. 

(4)  Develop  automated  system  for  handling 
data . 

It  is  recommended  that  the  program  for 
development  of  a  Navy  PPE  prediction 
methodology  be  implemented  and  that  a  "boot 
strap"  operation  be  started  to  begin  the 
development  of  a  data  bank  on  personnel 
performance. 
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The  AN/TSQ-73  Missile  Minder  was  a 
lightweight  mobile  automatic  data 
processing  command  and  coordination  system 
for  NIKE-Hercules  and  Hawk  Army  Air  Defense 
units.  The  AN/TSQ-73  integrates  radar  and 
identification  of  friend  or  foe  (IFF)  data 
from  vocal  and  remote  radars  for  console 
display. 


The  SAINT  model  of  the  AN/TSQ-73  system  was 
designed  to  simulate  the  tasks  performed  by 
a  single  operator/repairman  involved  in 
monitoring  and  operating  the  AN/TSQ-73 
display  console  during  a  simulated  mission. 
The  SAINT  model  was  comprised  of  four 
submodels:  operator  control,  aircraft 
control,  fire  unit  control,  and  systems 
control.  These  four  submodels  operated 
relatively  independently  of  each  other. 
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This  report  details  the  development  of  an 
integrated  package  of  computer  routines 
designed  to  aid  the  system  design  engineer 
in  determining  the  impact  of  the  human 
operator  on  system  performance.  The 
objective  of  SAINT  is  to  provide  the 
necessary  tools  and  conceptual  framework  to 
develop  simulation  of  complex  man-machine 
systems.  SAINT  enables  the  designer  to 
imput  a  description  of  the  activities  which 
the  human  operator  must  perform  in  the 
course  of  a  mission.  These  activities  are 
represented  in  a  task  network  framework  in 
which  task  performance  descriptions  as  well 
as  precedence  relations  among  tasks  are 
defined.  SAINT  performs  an  analysis  of  the 
task  sequence  and  provides  information 
concerning  operator  workload,  task 
completion  times  and  other  performance 
measures. 

The  SAINT  II  package  (an  expansion  of  SAINT 
I)  has  techniques  which  enable  the  user  to 
model  continuously  changing  variables  such 
as  aircraft  position  in  space,  engine 
temperature,  fuel  consumption,  etc. 
Additional  refinements  have  been  devoted  to 
the  incorporation  of  the  capability  to 
modify  operator  and  system  characteristics 
as  a  function  of  mission  contingencies  and 
external  events. 
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5.4  Conclusions  and 
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Each  operator  involved  in  a  mission 
simulation  with  SAINT  II  has  certain 
attributes  which  must  be  assigned  to  him. 
Using  input  data,  the  program  sets  seven 
operator  attributes  into  a  packet,  although 
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